Emulator vs Simulator vs Real Device

Testing Mobile Apps on Emulators vs Simulators (Virtual Devices) vs Real Devices

13 min readMar 14, 2024

Real devices come with a great value and benefits, but also with their own costs. Mobile simulators and emulators can offer valuable benefits for both developers and testers, but it’s worth noting that they also have limitations. Let’s compare and understand different mobile devices that testers deal with on a regular basis.

Emulators and Simulators

Simulation and emulation are frequently confused, but they are distinct concepts. Both involve creating a virtual replica of a real-world system, but there are significant distinctions between the two when it comes to mobile app testing.

In mobile testing, an emulator is a software that mimics the functionality of a real Android device and allows to test an app on it. It mimics lots of the hardware and software/OS features of the device for the production environment.
A simulator, on the other hand, is a virtual device for testing iOS apps on a Mac host, such as mimicking the OS behavior of an iPhone or iPad. Simulators mimic the basic behaviors, variables, and configurations of a real device that exist in an app’s production environment. Simulation involves replicating elements of the physical world within a virtual setting, allowing for an approximation of their functionality. It captures essential features, but may not fully adhere to the laws of the actual environment.

Emulators provide an operating environment that closely resembles the functionality and capabilities of a mobile device model, like:

Emulated incoming phone calls and text messages.
Various network speeds.
Location services
Device orientation and rotation.
Hardware sensors.

But the Android emulator doesn’t include virtual hardware for the following:

Bluetooth
NFC
SD card insert/eject
Device-attached headphones
USB

Simulators can be useful for early testing of iOS apps by creating a simulated OS environment that mimics the functionality of a specific mobile device model. However, it’s important to note that simulators have limitations and should not be relied upon for testing of hardware components, gestures, or real-world user scenarios. They are best for lightweight functional testing.

The following hardware is not supported in a simulator:

Ambient light sensor
Audio input, except for using Siri by choosing Hardware > Siri.
Barometer
Bluetooth
Camera
Motion support (accelerometer and gyroscope)
Proximity sensor

Due to their limitations, emulators and simulators are useful at an early stage of the development process in order to obtain feedback about the implemented features.

Emulators vs. Simulators

An emulator is a desktop app that virtualizes many aspects of a real device, including both the hardware and OS. It translates the instructions of the compiled app source code so that the app can be executed on a desktop computer. The emulator acts like the mobile device hardware and operating system, thus allowing the developer and tester to debug or test the application. Since the app is executed on a computer, not all of the mobile-specific hardware elements such as the sensors or touch gestures can be emulated.

Simulators mimic the OS, not the hardware. They allow the user to validate application flows, but not in a true production environment. They are less complex software applications that simulate a small subset of the device’s behavior. In contrast to emulators, simulators are only similar to the target platform and simulate the real device, making them much faster than emulators. It is also not possible to test device-specific hardware elements with simulators.

Differences between Virtual Platforms

Eran Kinsbruner presented the above table at a 2021 webinar. He drew attention to a couple of areas as examples.

Android emulators support .apk files. We can run the same binary .apk file on both virtual emulators and real devices. This is not the case on iOS. On a real Apple device, we can install an .ipa file. But when we test on simulators, we can only install a .zip or .app file, which is the dev-provided version of the app. It’s not a binary we’re going to deploy to an app store. This in itself exhibits that we aren’t going to end the journey of testing just with virtual devices, which will get us to a certain point. then we need to switch to the official binary that we’ll test on real devices.
As for the sensors, Android emulator supports the camera through instrumentation. But iOS doesn’t fully support face recognition on a simulator. The simulator just simulates the outcome of a correct and a failed face recognition, just like it does with Touch ID. It does not recognize faces. Check out my earlier post on automating the biometric functions with Appium on iOS simulators.
There are different MDM limitations between the two platforms.
Calls/SMSs/2FA are not supported on iOS simulators, because it’s an .app file (not .ipa), which is limited.
Security is only used on simulators. But again, we’re dealing with an .app, not .ipa, file. So we aren’t really covering the security aspect of the binary that’s going to live in production.
The operational architecture differences — ARM vs x86 chipset, which accordingly require us to compile the app differently.
The memory is, of course, different by nature of the architecture.

The most obvious difference between a simulator and an emulator is that a simulator attempts to duplicate the behavior of the mobile device, while an emulator tries to duplicate the entire inner architecture of the mobile device and is therefore closer to the target platform. In other words, an Android emulator mimics both the device’s OS and hardware — it’s an actual VM hosted on our computer. An iOS simulator only mimics the OS, not the hardware — it’s merely another app installed on our machine which we can open with command open -a simulator.

Mobile testing requires movement and different hardware, meaning that we need to test our app on physical devices to be sure that everything works together in real-life situations. Emulators and simulators should be used only for very basic tests such as simple functionality (is the button clickable?) or to make sure the look-and-feel of the app is OK.

Real Devices

Using real devices in mobile testing enables us to test our app on actual hardware, providing an accurate representation of how it performs in customer’s hands. This is useful for ensuring the app’s functionality and compatibility on various devices.

Real Devices vs. Emulators vs. Simulators

The discussion surrounding the use of real devices, emulators, and simulators for mobile testing has been a longstanding and contentious topic in the mobile industry.

Emulators and simulators are best for testing in the early stages of development. Emulators are more suitable for testing external behavior, and simulators for internal behavior.

Testing on real devices provides the most reliable results and is ideal for usability and performance testing. It also allows for more realistic testing by applying real-world conditions and user scenarios.

The most effective approach to mobile app testing is to use a combination of tests, including emulators, simulators, and real devices, depending on the stage of development.

In the early stages, when features are still being developed, it is beneficial to perform smoke tests, unit tests, and other types of testing on emulators from the developer environment. As the build progresses and the need for more comprehensive testing and quality assurance increases, it is best to run full testing on real devices, and also include real-world user conditions to achieve a more accurate testing experience.

Emulators and Simulators — Advantages & Use Cases

Simulators and emulators are both virtual mobile devices that offer various benefits for testing apps. Emulation and simulation play a crucial role in the software development life cycle, as they can greatly reduce costs for developers and testers. They are often faster to establish and run, have fewer errors, and are frequently integrated into the developer’s environment, making them convenient for quick feedback.

Advantages:

Variety

Simulators and emulators can virtualize a wide range of devices and operating system configurations, making it easy to test on multiple platforms, including those that are only supported through virtual devices. This allows for efficient validation and testing on specific device/OS combinations.

Price

Virtual device solutions, whether local or cloud-based, are significantly more cost-effective than using real devices. As development teams need to test early and at a larger scale, the cost of using real devices for every developer’s pre-commit validation can become prohibitive. Virtual devices provide a cost-efficient solution for organizations to scale and execute testing at an earlier stage in the process.

Starting on a Baseline

Using virtual devices allows for a consistent starting point for testing, whereas achieving the same on real devices can be time-consuming and require a factory reset. This consistency and predictability of the virtual device state increases the reliability of automated testing, and eliminates the possibility of unexpected issues such as a locked device by another user. This makes the test execution more consistent and reliable as the device is always in the same state as it was designed for the tests.

Use Cases:

Local Development and Validation

Developers and testers commonly use simulators and emulators on their local machines for development, debugging, and local testing. The integrated development environments (IDEs) for native mobile applications, such as Xcode and Android Studio, include virtual device tools as part of the standard installation. Both have become stable and feature-rich in recent years, offering a wide range of capabilities for advanced testing.

Continuous Integration Testing

Virtual device labs are mainly used for continuous integration (CI) testing. As DevOps and Agile methodologies become more popular, teams are testing earlier in the development process, known as shift-left testing. New test automation frameworks that are more aligned with developers’ skills and tools, such as Espresso and XCUITest, enable development teams to increase their test automation coverage. To run these tests, a proper execution environment, such as a test cloud, is required. Automated tests can be run either in the pre-commit phase, providing fast validation before committing or merging code, or triggered through CI several times a day, providing quick feedback to development teams on recent code changes.

Real Devices — Advantages

Nothing Compares to the Real Thing

While virtual devices can be useful for basic testing, truly validating an app’s performance requires testing on real devices. Virtual devices are helpful for functional testing, but they may also produce false positives, where the tests pass but the app may have issues in real-world scenarios. This is because simulators and emulators primarily test the main flow of the application (happy path), rather than how it handles negative or extreme cases.

Better User Interface (UI) Validations

To ensure the accuracy of the user interface, validation should be performed on real devices. Additionally, usability issues are more readily apparent when testing on real devices, as opposed to virtual devices. This is particularly true when inputting data via the keyboard, as real devices overlay the app while virtual devices present the keyboard alongside the device interface.

More Accurate Performance Testing

Real devices offer more accurate and dependable measurements of transaction times for performance testing. Additionally, the use of specific hardware in real devices can affect performance differently than virtual devices, which may also render the user interface differently.

Improved Hardware and Sensor-Related Validations

Common scenarios that can be virtualized include those that involve interactions with device hardware and sensors, such as the camera, accelerometer, and biometrics. However, it’s worth noting that in some cases, the behavior of real and virtual devices may vary.

Balance Virtual and Real Devices

**Mobile Test Pipeline**: Map coverage needs to your test pipeline. Divide a common pipeline by testing types and triggers. Virtual devices are often best used in earlier phases, real devices — in later phases.

As testing occurs throughout the entire SDLC process, from early stages to deployment and monitoring, it is necessary to utilize both real and virtual devices. A balanced approach that includes both types of platforms is essential for each phase in the application lifecycle.

A proposed approach would be to use a combination of virtual devices for early testing and real devices for later stages, such as usability and performance testing, and for validating the application in real-world scenarios. This will allow teams to take advantage of the benefits of each type of platform and ensure the best possible results.

SDLC Phases/Stages 🌕🌑 — CI/CD Flow

🌓 Code

Local Validation

During the development phase, developers should validate their code using either a local device or a virtual device, which is often integrated into the developer’s IDE. For UI validation, it’s recommended to use real devices rather than virtual ones to ensure that the outcome appears as expected.

Pre-Commit Validation

Unit and UI unit tests should typically be run on virtual devices (90% of the time). However, there may be instances where developers make changes to components that they anticipate will behave differently on real devices. In such cases, it is recommended to provide developers with on-demand tools that allow them to choose which tests to run and on which platform (virtual or real) to run them. This will enable developers to verify the behavior of their changes on both virtual and real devices and make more informed decisions.

🌔 Build

Commit Job

This process should run every time a developer makes a commit, performing a basic check to ensure that the change doesn’t break the build or cause significant regression. Because of the need to run this job at scale with each commit, these tests should be run on virtual devices to ensure efficient execution.

Multi-Merge Validation

These jobs validate the last X commits and check if one of them broke the build or caused a regression. This type of job usually runs every few hours or after a certain number of code commits. It runs more tests to provide broader coverage. In this stage, more real devices are introduced into the CI process, typically a mix of 70% virtual devices and 30% real devices should be used. This will provide a balance of efficient execution and realistic testing on real devices.

Night Job

This job runs regression tests and expands coverage. In this stage, tests are run on real devices to ensure the application’s performance in real-world scenarios.

The frequency of the CI process will vary depending on the team’s maturity level, it can run one or more times a day. The aim is to create a balance that enables scalability while still providing comprehensive test coverage and insights into the end-user experience. To achieve scalability and enable parallel validation for multiple developers and teams, a typical CI mix might consist of a combination of virtual devices for functional testing and real devices for usability and performance testing. This will allow for efficient execution and provide valuable feedback on the application’s overall performance.

🌖 Test

Additional testing activities that take place outside the CI process will be run on real devices. The objective of these tests is to validate the overall end-user experience, functional flow coverage, UI validation, and performance testing. All tests (100%) should be run on real devices to ensure the best possible results.

🌗 Monitor

While some organizations use Real User Monitoring (RUM) solutions that run on real devices, synthetic monitoring, which involves executing single-use automated tests to measure app performance every 10–15 minutes, also has value. For the most accurate results, it is recommended to use real devices for synthetic monitoring. This will provide the most accurate representation of the end-user experience and performance of the app.

Takeaway

In a nutshell, in the pipeline itself the cost of a bug grows as we shift right. Hopefully, a bug is not found in production or any time close to the release, because at that point the bug becomes expensive. We need to track down the root cause, figure out what’s going on, and do lots of regression testing to assure the bug fix did not harm any other aspects. This is why when we can test early, virtual devices on the mobile front can help us shift left and test early. This is where we can really catch a lot of the early stage bugs and reduce the overall cost of the app going out to production and having an escaped defect that might be found prior to the release.

Effective implementation of continuous testing and achieving optimal velocity in the mobile development process requires a balance between simulators, emulators, and real devices. Each of these platforms offers distinct values and advantages to developers and testers, but these benefits can only be fully realized when used at the appropriate stage of the development life cycle.

Proper planning of test coverage, the platform under test, and testing tools is crucial for continuous testing activities. It’s also important to stay informed about new devices, simulators and emulators that are released in the market. It’ll help you design the ideal mix of devices for your testing strategy and make the most of the benefits of each testing environment.