Cross-Browser Testing, what’s that for?

Did you just come up with this new brilliant web app idea and you are now wondering how to make sure it works on all devices? or do you already have an up & running web app but your users are tired of reporting new browser-specific issues everyday?

In both cases, this post is for YOU!

Today, we, web users, want to browse the same web on our laptops, tablets, phones and watches. If your web app works great on Chrome desktop but is partly broken everywhere else then you might probably lose 60% to 80% of your audience!

Now, you are probably thinking “but, with HTML 5 standard isn’t it just the same on all devices? Am I not safe with abstraction tools like AngularJS / Angular Material / Bootstrap / Less/ Sass?”. Well, that’s what we were naively expecting when we started Wishtack and we were surprised to discover all the variations between browsers and the related issues.

Just to name some:

Flexbox wrapping issues with Safari & Internet Explorer. It’s currently impossible to use flexbox wrapping with a list of flex items. Elements must have fixed dimensions.

Flexbox performance issues on Firefox. On Wishtack, flexbox was causing a thousand times more performance issues than AngularJS. (https://css-tricks.com/does-flexbox-have-a-performance-problem/)

In some conditions AngularJS’s “ng-submit” is broken on IOS since IOS 8.0.

Safari loses URL fragment on redirect. A 6 years old bughttps://bugs.webkit.org/show_bug.cgi?id=24175. Happy birthday!

The way browsers handle relative element’s height differently.

Cross-browser testing is thus essential for you web app’s success!

Browsers & devices you should prioritize in your Cross-Browser Testing

There are hundreds of devices and browsers out there and you probably can’t test them all so you’ll have to choose those who cover most of your audience.

If your web app is not live yet then you can use public stats like this onehttp://gs.statcounter.com/#mobile_browser-ww-monthly-201404-201504.

Otherwise, we recommend to generate this list using your current audience statistics.

This can be achieved with Google Analytics:

On the side menu, go to “Audience => Technology => Browser & OS”.
Choose “Operating System” as primary dimension.
Choose “Browser” as secondary dimension.

And you should get something like this:

Or if you are using Splunk (download it for free onhttp://www.splunk.com/en_us/download.html or run it directly using Amazon AWShttps://aws.amazon.com/marketplace/pp/B00GIZITUO):

Install the following add-on: https://splunkbase.splunk.com/app/1843/
Copy the following search query, replace “user_agent” by your user agent field and run it on Splunk:

| search *
| lookup user_agents http_user_agent as user_agent
| replace "unknown" with "" in ua_device, ua_os_major, ua_os_minor
| replace "Windows*" with "Windows" in ua_os_family
| eval user_agent=ua_os_family + " - " + ua_family
| regex user_agent!="(unknown|PhantomJS)"
| dedup address user_agent
| eventstats count(address) as total
| stats count(address) as count by user_agent, total
| eval percentage=(count * 100/total)
| table percentage user_agent
| eval percentage=round(percentage, 2)
| sort -percentage
| streamstats sum(percentage) as cumulative_percentage

We didn’t include OS and browser versions in order to reduce the list’s length but we recommend adding them to know which versions to prioritize in your cross-browser testing.

| search *
| lookup user_agents http_user_agent as user_agent
| replace "unknown" with "" in ua_device, ua_os_major, ua_os_minor
| replace "Windows*" with "Windows" in ua_os_family
| eval user_agent=ua_os_family + " " + ua_os_major + " - " + ua_family + " " + ua_major
| regex user_agent!="(unknown|PhantomJS)"
| dedup address user_agent
| eventstats count(address) as total
| stats count(address) as count by user_agent, total
| eval percentage=(count * 100/total)
| table percentage user_agent
| eval percentage=round(percentage, 2)
| sort -percentage
| streamstats sum(percentage) as cumulative_percentage

As you can see, by testing the most popular os / browser combination, which in our case is a “Windows – Chrome 41”, we would only cover 21.57% of Wishtack audience.

By adding latest versions of other popular combinations like “IOS – Safari”, “OS X – Safari”, “Windows – IE”, “Windows – Firefox” and “Android – Chrome”, we would cover 74.42% of Wishtack audience.

In theory, there shouldn’t be any big difference between major versions of browsers like Chrome 40 and Chrome 42 but it mainly depends of the browser features your web app is using. In that optimistic case, as you can see on the first table, we should have a cross-browser testing coverage rate of 89.11%.

Responsiveness Testing

Now that you have selected the browsers to include in your cross-browser testing, you probably want to know if your web app’s responsiveness is doing well.

There are at least four solutions:

Buying IOS and Android devices of different sizes.
Using google chrome’s device mode:https://developer.chrome.com/devtools/docs/device-mode
Using Browserstack.
Run your automated tests with different resolutions. (see Cross-browser Test Automation)

Empty your pockets or simply use Browserstack

As you are probably expecting, buying all the devices you want to test can get really expensive and you will at least need the following:

MacBook (for OS X testing): $1200
iPhone 6: $650
iPad: $500
Nexus 5: $450
Nexus 7: $200

This makes a total of $3000 per tester and you’ll have to upgrade your gear every year. Plus, you are not covering that much of your audience with this.

That’s when Browserstack comes to the rescue!

Browserstack is a cloud service that allows you to connect instantly to:

OS X and Windows virtual machines with hundreds of os and browser versions combinations.

Android devices emulators.

IOS devices simulators.

And even real android and IOS devices controlled using Appium.

They have an attractive pricing that starts at $29 per month and which gives you access to all these browsers and devices in just one click.

Live Testing with Browserstack

Using Browserstack Live, you can access any kind of desktop browser or mobile device from your browser then test any web application on Internet.

But what if you want to test your local or intranet web app?

Live testing local or private web app with Browserstack

In order to test your local or private web app with Browserstack, you just have to install the “browserstack local testing” chrome extension and it will create a tunnel from the remote virtual machine on Browserstack to your local machine, thus allowing you to test any local or private web app on any browser.

Browserstack Cross-Browser Screenshots & Responsive Testing

Testing just one URL on 20 browsers can take a lot of time so Browserstack came up with the Screenshot & Responsive feature. This feature allows you to enter a URL that will be tested on the browsers you select then it shows a screenshot for each of them.

Cross-browser Test Automation with Protractor and Browserstack

Testing using Browserstack Live is fun but manual cross-browser testing is time consuming, boring and unreliable. How many testers would you need to test all the corners of your web app, on every browser and after each release? Wouldn’t that be the best way to make the testers quit the company after one week?

If you have read this post http://www.blog.wishtack.com/#!The-Anti-Startup-Experience/cuhk/C417E006-A622-4365-89CB-5D9EF826EC85 then you probably know that boring tasks are forbidden at Wishtack 😉

That’s where our favorite Browserstack features gets into the game, Automated Testing!

With Browserstack Automated Testing, either you have an AngularJS or a non-AngularJS web app, you can write your tests using Protractor (or directly manipulate the Selenium Webdriver in your favorite language https://www.browserstack.com/automate/python) and run them on the browsers of your choice.

Try it now

For this great occasion, we wrote some open-source modules that you can integrate into your project.

If you are starting from scratch, you can clone our boilerplate projecthttps://github.com/wishtack/wt-protractor-boilerplate, write your Protractor testshttps://angular.github.io/protractor/#/tutorial, and run them on Browserstack.

If you already have a git project, you can merge it with our boilerplate.

And finally, if you are already using Grunt or Gulp or Protractor, you can directly usehttps://github.com/wishtack/wt-protractor-runner and https://github.com/wishtack/wt-protractor-utils.

For instance, our boilerplate comes with a non-AngularJS demo test called “test-google-wishtack” that simply googles Wishtack on different browsers and devices.

describe('google wishtack', function testGoogleWishtack() {

    var _waitForElement = function _waitForElement(selector) {
        return browser.driver.wait(browser.driver.isElementPresent.bind(browser.driver, selector), 10000 /* Timeout. */);
    };

    it('should return wishtack', function shouldReturnWishtack() {

        var PageGoogle = require('./pages/page-google');

        var pageGoogle = new PageGoogle();

        /* Go to google. */
        browser.driver.get(pageGoogle.pageUrl());

        /* Wait for form. */
        _waitForElement(pageGoogle.selectorInputSearch());

        /* Search. */
        browser.driver.findElement(pageGoogle.selectorInputSearch()).sendKeys('wishtack');
        browser.driver.findElement(pageGoogle.selectorForm()).submit();

        /* Wait for result. */
        _waitForElement(pageGoogle.selectorResult());

        /* Check result content. */
        expect(browser.driver.findElement(pageGoogle.selectorResult()).getText()).toMatch(/www.wishtack.com/);

    });

});

1. Just install node & git.

2. Clone the project:

git clone https://github.com/wishtack/wt-protractor-boilerplate

3. Install gulp:

sudo npm install -g gulp

4. Create a browserstack account, set your BROWSERSTACK_USER andBROWSERSTACK_KEY env vars then run the “test-e2e” task:

BROWSERSTACK_USER=... BROWSERSTACK_KEY=... gulp test-e2e

And you should get something like this:

os=OS X Yosemite, device=, browser=chrome: SUCCESS (1 test, 1 assertion, 0 failures).
os=OS X Yosemite, device=, browser=safari: SUCCESS (1 test, 1 assertion, 0 failures).
os=WINDOWS 8.1, device=, browser=chrome: SUCCESS (1 test, 1 assertion, 0 failures).
os=WINDOWS 8.1, device=, browser=internet explorer: SUCCESS (1 test, 1 assertion, 0 failures).
os= , device=, browser=android: SUCCESS (1 test, 1 assertion, 0 failures).
os= , device=, browser=iPad: SUCCESS (1 test, 1 assertion, 0 failures).
os= , device=, browser=iPhone: SUCCESS (1 test, 1 assertion, 0 failures).
[13:44:24] Finished 'test-e2e' after 4.13 min

But what if we want to manually see what’s happening on each browser during the test?

Easy! You just go to https://www.browserstack.com/automate and you will see the browser instances history.

You click on the one you want then on “Visual Logs” and you can see the screenshots that were taken by Browserstack during the tests.

In addition to this, you can also watch the tests running live by clicking on “Live Interaction” button. This is also useful for debugging as you can pause the tests at the step you want using Protractor’s “browser.pause()”, open “Live Interaction” and take control of the browser then run your favorite virtual browser’s debugger.

Automate your Cross-Browser Testing and keep writing unit and integration tests!

As you can see, with the powerful combination of Protractor, Selenium and Browserstack there’s no excuse for not automating your cross-browser testing.

Of course, cross-browser end-to-end tests are necessary but not sufficiant, you still have to write unit and integration tests for your frontend and backend.

And don’t forget, Wishtack is here to make your wishes come true so don’t hesitate to get in touch with us if you need any kind of help, training or coaching.

Wishtack

Cross Browser Testing & Test Automation with Protractor and Browserstack

Cross-Browser Testing, what’s that for?

Leave a Reply Cancel reply

Wishtack

Cross-Browser Testing, what’s that for?

Share this:

Like this:

Related

Leave a Reply Cancel reply