Google Maps Vs Apple Maps Comparison Study

Our objective was to conduct a formal usability study on Google Maps and Apple Maps. Our research question was to find out which one of the two applications is better in terms of usability? Our hypothesis was that Google Maps is better in terms of usability.

Brief

Evaluation of Google Maps vs Apple Maps for iPhone users. A constant rivalry for the market share exists between Google Maps and Apple Maps. The aim of the project was to explore which one of the map applications is better in terms of usability, defined by the ISO standards. The evaluation was performed on an iPhone with iPhone users.

As users of the application, we quickly found out that the most important usability issues would be raised with this experiment, as these are the most used functions for both applications.

• Finding a location on the map
• Retrieving its opening hours
• Tracing the shortest route to the location
• Navigating to the location by walking

UX Group: Lucas Corea, Mikko Honkanen, Rafael Goldenberg, Yash Todar

Stages

Methods

Combinations used in the In-between group study

1. Controlled Experiment

2. Out in the Wild testing

3. Comparative usability study based on ISO (Effectiveness, Efficiency & Satisfaction)

We tested our hypothesis based on the definition of usability in the ISO standards

• To test the effectiveness of the applications, we will count the frequency of errors and measure the effort needed to recover from errors (depending on the function we count the steps or take the time the user needs).
• To test for efficiency we will measure how long it takes for the users to complete a set of tasks.
• To compare the satisfaction of the products, we will use a SUMI survey. The methods used for testing the effectiveness and the efficiency of the applications will give us quantitative data, while the SUMI survey will give us qualitative data.

Procedure

1. Subjects of both the groups were asked to individually accomplish the same tasks. Firstly, the participants were asked for their consent through the consent form. After that, the subjects had to complete the tasks list, while time and other stats were taken. They had to do a navigation task as the part of the study and once the navigation task was done, the SUMI survey was given to them to fill out.

2. Finally, the subjects were asked general and open ended questions about the product evaluated and the test itself.

3. The test location was done indoors at KTH Campus (building D and E’s quiet corridors) for the first three tasks, and then from a common point outside the main gate of the building to the task specified destination, which in this case was Jarla Trafikskola.

4. After the tests were done, the research team started to analyze the gathered data. General observations - either during the tests or while watching again their recordings - and feedback from the subjects generated qualitative data, as well as the number of occurrences for specific behaviors. Statistical analysis was done based on the time to accomplish each of the aforementioned tasks, the number of times the user deviated from the optimal path and the number of steps added when the user diverted from it.

This gave us quantitative data for precise comparison between the two applications. Since a quantitative method was performed to raise data for actual performance, the results of the SUMI survey were used to confront the actual to the perceived performance of both applications, which has led to a better conclusion for the research question.

For the evaluation test, several tools was used. Mainly, the subjects handled an iPhone running the latest version of each app. To measure how much time each task took to be accomplished, we used a timer, screen capture footages and finally, we also used a movie recording device in order to make observations after the test.

There were many alternative ways to accomplish such tasks, but we previously determined the optimal path and considered any deviation from it in the evaluations. Also, the time needed to complete the tasks - using such optimal path - was taken and compared between same tasks of the different interfaces. If the user deviated from it, the number of steps that are added to the completion of the task was counted and also compared between the two apps, as well as the time spent. We counted the frequency of errors and measured the effort needed to recover from errors, to measure the effectiveness.

Results 1

With the notes taken, and with the screen and video recordings in hand, the group gathered all the results from the experiment. The data collected was then categorized based on the ISO usability standards and a proper analysis was carried out as mentioned previously.

When approaching the destination, the application notifies of the arrival already within 10 to 20 meters from the actual spot. Secondly, the information given to the user by Google Maps is sometimes misleading or hard to see. Once the users were at about 50 meters distance from the destination, the interface changed to show information about the destination automatically. This was misleading for every user, as they interpreted this information as an arrival notification.

We measured the effectiveness by calculating the deviations from the optimal path. The order of the tasks as designed constituted the optimal path for performing them, although this was not mentioned to the users and they had the freedom to choose the order they would like to follow. Only two users - one from each app - deviated from the optimal path.

To evaluate efficiency, we measured the time for the users to complete the tasks. The time values for the indoor tasks for both applications were insignificantly different, as shown by the ANOVA test (p = 0.88). For the outdoor tasks, time values were also insignificantly different.

To compare the user’s satisfaction about the products, we used a shortened SUMI survey. A standard way for analyzing the SUMI survey exists, but wasn’t used in our case. This is due to our test not covering all aspects asked in the survey, so all questions of the SUMI survey could not be included. Therefore, we calculated the sums and means for each data dimension and additionally compared the results for each question, instead of using the paid standard SUMI analysis method.

Overall, the mean values for Google were better in all the dimensions, except for efficiency in which both user groups had the same mean value.

It is safe to conclude that for Google users the subjective views of the software were clearly better. Especially based on the Affect dimension the Google users find using the application more enjoyable.

When comparing the variance in the answers between the two user groups for each question, Apple test users rate the application worse than the Google Maps test users. This again proves that the users’ subjective view of Google Maps is better than that of Apple Maps.

Results 2

Effectiveness

Deviation from optimal path: two users - one from each app - deviated from the optimal path. A usability problem was found that once you trace a route it is not possible to retrieve information about the destination anymore.

Innacurate feedbacks

In Google, the arrival notification may be misinterpreted as if the user had already reached the place; in Apple, the location is misplaced in the map. Apple's user icon on the map is imprecise and doesn't give proper feedback of where the user is heading to. Apple's arrival notification lacks visibility.

Efficiency

Indoor timings: the measured times for the completion of the tasks indoor for both applications has appeared to be insignificantly different.

Recommendations

Based on the frequency of behavior during the evaluation we recommend both applications to be improved in the following way:

It is a bit difficult to give concrete recommendations for Apple Maps on how to improve the user satisfaction other than improving the accuracy and the optimization of the suggested routes.

The next step after this evaluation would be to extend the number of users used for the study to gather more data from a bigger sample size and get better results for quantitative as well as qualitative data.

As both apps are packed with many more different features and a greater number of users can still be tested for decisive conclusions on usability errors, we encourage that further usability testing should be taken.

For Google

• There should be no change of interface when the user is approaching the destination, as it is understood as an arrival notification and makes the users to stop navigation too early.
• There could be a pop up before reaching the place which should not take the user to another screen.

For Apple

• Place a pointer indicating the user’s current position, as the current icon is imprecise in informing the direction and position of the user.

For Both

• None of the applications allows the user to retrieve the destination's information in the navigation mode or when you trace the route, which should be revised. Some features like voice control were outside of the scope of our evaluation and could still be tested.
• As the newer versions of both of these maps will update current usability issues, which can further produce results to compare what issues were solved and what errors could still be existing.

Conclusion

The test users could complete the test tasks with a very small amount of errors. However, even though most users were able to use the applications without effort, some usability issues led to confusion.

To improve the user satisfaction, Apple Maps would need to improve the routing options, especially when starting the navigation. Both applications would also improve their usability, if the users could retrieve information about the destination while being in navigation mode.

The experiments conducted showed to be effective in comparing the two applications, as the data collected for the results, coupled with the analysis of the qualitative and quantitative data, raised concrete and conclusive information about the usability performance of both apps.

Our evaluation study indicated that both applications performed very similarly in the usability tests. However, the applications are not completely equally good in terms of usability, since the user satisfaction with Google Maps was better in our tests.

As the sample size of the study is small, further validations and more user studies are required and more detailed analysis is required to prove the hypothesis true.

The users in both groups use the applications for similar purpose. The most common use of the software is navigating to a certain place. One user in both groups also uses the application learning about the city and nearby places of interest. One Apple Maps test user uses the application for planning how to use the local transportation and one Google Maps user uses the map for looking at 3D maps.

Three Google Maps test users rate the kind of application as extremely important to themselves and three rated it as very important. Out of the Apple Maps users two rated the kind of application as extremely important to themselves, three as very important and one as not very important.

Five out of the six google Maps users rate their own software skills and knowledge as very experienced and technical while one test user rated I’m good but not very technical. Out of the Apple Maps users three rated themselves as very experienced and technical while three rated themselves good but not very technical. Overall the Google users rated themselves as more experienced and technical.

Learning outcomes: Advanced Evaluations, Field Testing, In-Between Studies, Comparitive Studies, Pilot Testing, SUMI surveys, Usability Studies