Austin Agile QA sig


I recently gave a talk to sum up my time as a mobile tester.  These are the notes I put together before the talk.

I joined HomeAway in June 2013.  At the time we had a native iOS app and a native Android app.  We were still building with Xcode 4.  This talk applies to native apps for iOS and Android only.


The technology I chose to do this was Appium.  I was new to mobile and had a Watir background testing web apps.  At that time the iOS world also had Calabash, Frank, UIAutomation, ios-driver and KIF.  The Android world also had Calabash as well as MonkeyTalk, Robotium UiAutomator(note, this looks like UIAutomation but is not) and selendroid.  So, only 2 libraries worked on both systems Appium and Calabash.  Calabash, like KIF, required compilation changes.  I did not understand our build process and was not able to get a the framework compiled in.  I preferred to use rspec over cucumber so I went wit Appium.

Appium in 2013 needed accessibility to be a viable testing framework.  That would later change for Android but at this time that was the only way to reliably look for elements on the screen.  Tests were first written in an ad-hoc way to prove out that buttons could be pushed, swipes could be done and text entered and verified from the screen.

After getting the more difficult iOS proof of concept up I was able to prove out Android.  At this point I started creating formal tests where both iOS and Android were using the same source.  I was writing tests for stories that both Android and iOS had both finished a while ago.  Both projects were supposed to work the exact same way. If the Android app didn’t work like iOS it was a bug.  With these two constraints (test existing code and both systems need to work the same way) I was able to test both systems with the same code.  There were a few if blocks of code in my view/page-object layer but it all worked.

When our project went to Xcode 5 I lost the ability to swipe in Appium on iOS. Without swipes to get past the first 3 pages of my application iOS and Appium just stopped working.

While iOS started going down in flames Android got better.  In the summer of 2014 I was able to use IDs and didn’t need to rely on accessibility information.  Shortly after that I was able to get my tests running for English and French.  We were running the full regression in under 45 minutes using the tests for a short period of time.  Maintenance of these tests was high but they did work and one person could keep them green while adding coverage at a slow pace.


Eventually, due to lack of iOS coverage, we moved to Calabash.  Calabash required code to be compiled into the iOS application but because this change came from the development organization QA and dev was able to work together and make this happen.  It wasn’t easy because our application used CocoaPods but it did happen.  Today it is easier to install.  This was all put back into Jenkins but the test automation team grew to 3 people at this point where we had a dedicated iOS and a dedicated Android developer.  Having one tester focus on iOS and a different person focus on Android helped quite a bit.  This is super important when you are testing currently developed features in different releases with different schedules.

When our team went to Xcode 7 I had problems getting the tests running in Calabash.  The problem came up with the new simulators that seem more like a new phone than the Xcode 6 simulators.  You see the Apple icon with a progress bar.  This reminds me of Xcode 5 and Appium again.  The issue may have been fixed but I rolled off of the mobile project and just just running tests with Xcode 4s simulators.

Take aways:

  • The closer you collaborate with your developers the better chance of success you have.
  • Going with vendor test tools seems like a good idea at this time.  If you don’t be prepared to cheerfully change technology at some point in the future.
  • Don’t fall in love with the idea of one set of tests that run on both iOS and Android, it might work but has a lot going against it.
  • You need fast reliable tests don’t be tempted to go after coverage and ruin that.
  • The closer you collaborate with your developers the better chance of success you have.
  • Context switching between iOS and Android is rough, testers dedicated to each platform helps.
  • Keeping a meaningful set of tests green and relevant on mobile isn’t easy.
  • Testing your mobile APIs is a big win





Calabash doesn’t work with well with two pop-ups at the same time

In my app, when a type of user logs in, you will see a permission pop up asking to send push notifications.  You will also get an alert that there is a newer version with better security features.  Calabash will automatically dismiss the push notification message.  The other alert is up to me to dismiss.  However, 4 different things can happen here:

  1. Usually, after waiting for the push to go away I can dismiss the other pop-up
  2. Sometimes I get a long pause and an exception trying to touch
    uia action failed because: Timed out running command uia.tapOffset('{:x 92.5, :y 358.75}')
  3. Sometimes this makes the app crash, but I can’t reproduce it without Calabash
    Connection refused - connect(2) for "localhost" port 37265 (http://localhost:37265) (Errno::ECONNREFUSED)


     Could not parse response ''; the app has probably crashed (RuntimeError)
  4. Other times it doesn’t go away and the next wait_for_none_animating will throw an exception because there is a pop-up
    Timeout waiting () for condition (NONE_ANIMATING) (Calabash::Cucumber::WaitHelpers::WaitError)

I don’t have a good solution to this problem other than, don’t do that, two popups at the same time is bad UI.

Jenkins agents are hanging with a security pop-up

I probably tried 10 different things to make the security pop up described below go away

The problem is that it takes a day to see if you fix actually works and your CI runs are dirty because it requires human intervention to get it running the following day.  I was disabling CI when I went home and enabling it when I fixed the agents the next morning.  The solution that worked for me was near the bottom.  On Yosemite 10.10.5 the command that worked is:

$ DevToolsSecurity 
Developer mode is now enabled.

Do not put spaces in build paths

Use a dash or underscore instead of a space in your Jenkins job names.  When you put a space in them you get spaces in the workspace.  While it is annoying to deal with this directory name on the command line you can get by.  Most of your code will be fine, but some shell scripts may not work.  We recently moved from Crashlytics to Fabric pod and some file script is derived data. The space in $PODS_ROOT broke this:

${PODS_ROOT}/Fabric/Fabric.framework/run some_guids

Another benefit is that URLs for you jobs will not have %20 sprinkled in them.

Calabash sees elements behind the keyboard but itsatrap

If you are filling in text fields the iOS soft keyboard will pop up, when you want to interact with the next element Appium will attempt to click into the field which may result in a keyboard key being pressed.  If you are using enter_text then the text that was intended for the field in the query string will then go into the field with current focus.

With my app the keyboard action key, Done, attempts to submit the form.  There is a next key but it goes to the next field that involves the keyboard.

touch "toolbarButton marked:'Next'"

The solution I ended up with was to write a function that would scroll the bottom of the element above the top of the keyboard

  def scroll_until_above_keyboard element, scrollable, options={}
    timeout = options[:timeout] || 5
    max_y   = top_of_keyboard

    wait_for(timeout) do
      if query(element).empty? || bottom_of_element(element) > max_y
        scroll scrollable, :down

  def top_of_keyboard element="toolbarButton marked:'Done'"
    rect = query(element)[0]['rect']
    rect['center_y'].to_f - rect['height'].to_f / 2.0

  def bottom_of_element element
    rect = query(element)[0]['rect']
    rect['center_y'].to_f + rect['height'].to_f / 2.0

This is the function

until_element_exists(page.first_name_field, :action => lambda { scroll(page.webview, :down) })
enter_text page.first_name_field, model[:first_name]

scroll_until_above_keyboard page.last_name_field, page.webview
enter_text page.last_name_field, model[:last_name]

scroll_until_above_keyboard page.email_field, page.webview
enter_text page.email_field, model[:email]

scroll_until_above_keyboard page.phone_field, page.webview
enter_text page.phone_field, model[:phone]

csa_query = page.csa_option model[:csa]
scroll_until_above_keyboard page.csa_, page.webview
touch csa_query

2015-09-11 what I figured out this week

Sometimes a page does not load in the order that it is displayed or the content you think is on the screen is scrolled off the top.  In this case the second edit button was being pressed before the first one was showed up.  I was pressing the payment schedule instead of the quote.  The fix was match only one button.

touch "* marked:'Quote' parent * index:0 descendant button marked:'Edit'"

instead of

touch "button marked:'Edit' index:0"

iOS Simulator Screen Shot Sep 8, 2015, 1.23.02 PM

The second time I visit my search page I cannot enter text:

enter_text page.search_field, ‘miami’;done;wait_for_none_animating;touch page.search_button
RuntimeError: uia action failed because: Timed out running command uia.elementWithKeyboardFocus().setValue(“miami”)

After it times out if you leave the simulator up you will eventually see ‘miami’ show up in about 4 minutes.  Using the option use_keyboard: true does