Testing is a decades-old, broad, and fast-growing space equipped with the agility to constantly subsume new processes, tools, and methodologies. As most of my experience comes from testing with hardware, I used my knowledge to discover key steps that accelerate achieving high-quality outcomes.
This blog brings a consolidated list of the fundamental steps for testing with hardware that you should follow when working on complex projects.
Testing with hardware, what does that mean?
Projects that require testing with hardware include:
- a piece of hardware
- firmware that is running on hardware
- software that communicates with the hardware
- a communication protocol between the hardware and software.
These projects can be rather complex. To help you get a clear picture of what this process looks like, what steps it entails, and how to overcome the hurdles, let’s look at the typical scenario, from start to end, focus on the critical points in the process, and offer solutions to each of them. Ideally, this should be the desired flow of the events that should lead to a successful deployment to the end customer.
1. Close the (domain) knowledge gap
The first obstacle you may come across when starting the project is a potentially huge domain knowledge gap between your current knowledge and the demands of the project. So, you eventually realize that you need to start quickly closing this gap. Here are a few sources you can leverage to get what you are looking for:
- Colleagues — Reach out to your colleagues to hear expert advice from someone who has already been involved in a similar project or is working on one currently.
- Customer — Usually, the client has more extensive knowledge about the business domain than you.
- International standards —These kinds of projects sometimes must be compliant with certain international standards. They might be a bit complex at first sight but will generate a lot of critical questions and provide answers.
- Certification bodies – Certification documentation represents the source of extensive knowledge you can further leverage for your future projects’ success.
2. When hardware is missing, emulate it
Another thing you learn quickly is that the hardware is elusive. From my own experience, we were usually able to start the development of a software piece of the system before hardware was ready. To make things more complex, it usually turns out that it takes 3 to 6 months, sometimes even a year, for hardware to be completed. So, what can you and your team do? You can build an emulator. There are a few important things to keep in mind here:
- Emulators are complex and usually mimic the behavior of real hardware (device).
- The emulator is a piece of software you will have to maintain.
When thinking about building emulators, consider making them universal. So, if you have multiple instruments or devices, think about building one emulator, as it will enable you to manage everything in one place. Also, remember that the emulator is another piece of software with its own sets of expectations and requirements, which don’t necessarily have to come from your customer — they can also come from your team. Either way, it is important that you treat it as a separate piece of software.
3. Advocate for testability
Once the emulator is in the house, you are slowly closing the domain knowledge gap, and the initial parts of the software are becoming available for you to test. For these kinds of projects, it is imperative to advocate for testability from the very beginning.
Projects of this kind usually do not have a UI part (or it is developed in the later stages), so you should opt for testing that does not include this part, at least initially. In this case, we usually develop analytics engines, calculation engines, and data transformations first. Although you should ensure that the app is testable from the very beginning, you need to shift your focus to controllability and observability because it is crucial for controlling the system, bringing it into the right state easily, and observing what’s going on in that state.
So how are you going to do that?
- Provide the right inputs at the right time — Define the right scope and the right sample size from the very beginning because it will verify the very foundation of the product you are building. By making it so, with the help of an emulator, you will be able to control the application and achieve the desirable states.
- Creating an efficient logging strategy — Since you are testing a lot of things that are going without the UI (calculations, transformations, data flows), you should be able to find whatever you need rather quickly and more efficiently. So, make sure your logging strategy works for you by staying focused on what you need there and avoiding unnecessary information.
4. Automate what steals your time
Now that you feel confident because you are closing your domain knowledge gaps — you have the emulator, which is helping us test various features, and you are able to control the system and have a clear overview of what’s working or not.
Everything seems to be running smoothly, right? Not really!
You suddenly realize that the level of complexity of these systems is much higher than you think. No matter how much you prepare yourself for the unexpected turn of events, you can never know what challenges are lurking in the dark. The devices and instruments can run in different configurations, and, in the end, you need to test them all. All of this adds up to the complexity.
And what do you do in this case?
Automate whatever steals your time.
When it comes to automation, projects like these are a bit specific. For instance, let’s say you are measuring blood cholesterol levels. Creating a result with some input data, preparing that data, analyzing what happened with it, and realizing that your cholesterol level is 4, takes a lot of time. To avoid this scenario, you should automate what takes most of your time, including data preparation, data transformation analysis, or analysis of end results.
At this point, the real hardware is usually ready, and you can start working on a real thing and play with it finally. Now, the question arises:
Emulators are good, but should you trust them?
Emulators are good, but should you use them now that you have got the real thing? In most cases, you will not stop using them. As a matter of fact, you will probably use them more than expected. My team used to continue using emulators even when we had the hardware or the prototype ready — emulators are simply more convenient and faster. However, there are cases when you must play with the real thing because emulators do not have the required capacity to perform the needed activities.
Here are a few things emulators cannot do:
- Time-related testing
- Performance
- Reliability
I’ll share one success story where my team managed to use hardware as the ultimate emulator. They were working on a new product built from scratch, meaning that hardware, firmware, software, and communication were all built from zero. The firmware was built in a way that it could be flashed onto a device or run as a separate (desktop) application. This means it was running as an emulator independently of an instrument. Overall, by designing a firmware that is also an emulator, the team saved a lot of time and energy because it allowed them to update firmware and the emulator simultaneously, without having to investigate potential issues separately.
5. Test in real conditions
One thing you need to remember is that sometimes the equipment and the devices you are working with might be running in special conditions. That’s why it is important to test your devices in real conditions, with the right environment, temperature, humidity, pressure, or, at least, to mimic the conditions and make the environment as authentic as possible. This way, you can have a clear view of how your equipment behaves in real-world circumstances.
6. Automate certification
There may be projects where you need to pass certain certifications issued by third-party certification bodies. If you need to do this often, you might want to consider automating the certification of the process. With some projects, you need to pass certification with every release.
At its core, certification is a process where the certification body issues a set of conditions that your system needs to satisfy. To be able to pass the certification, you need to do a set of tests which might be tiresome and time-consuming with every release. However, the good thing is that certification bodies usually accept log files or similar input as proof that you executed a prescribed set of conditions. So, when building a logging strategy, you should also think about easy ways to display them in your log files.
7. Make deployment smart
Finally, you have reached the point when your project is ready for deployment. But you need to make it smart. Make sure you do staged rollout and add auto-update functionality to your system if possible. The auto-update feature on systems like this is still rare. Still, if you see an opportunity to add this feature, do it because it will save loads of time and energy for technicians and people on the customer side.
Testing with confidence
Although there is no magic recipe for each project when testing with hardware, as every project is unique, my experience has taught me that these seven steps have helped my team find solutions to some of the most unexpected and challenging problems we’ve encountered along the way. I expect these principles help teams drive testing strategies even as they venture into newer technology territories to achieve high-quality results.
Want to learn more about how to streamline testing processes? Let’s talk.
