How Wirecutter Vets the Security and Privacy of Smart Home Devices

Smart-home devices allow you to light up dark walkways after a late night, fire up the AC on your way home from the airport, and peek in on pets during the workday. Despite all of that comfort and convenience, for many people there remains an undercurrent of distrust—some of it justified—when it comes to their smart-home devices. We worry, too, which is why we’ve made security and privacy a focus of how we test devices and make recommendations.

In order to recommend the right devices, we look at features, functionality, pricing, and even aesthetics. However, for more than a year we have also phased in security and privacy testing for every smart-home guide we publish. And we are adding a comprehensive security and privacy section to every smart-home guide we publish by the end of 2020. Here’s how Wirecutter staffers vet all of our smart-home picks.

We do research

Every Wirecutter guide starts with a scouting report. We look at common features, pricing, and performance, but we also dig in deeper to look for any potential security and privacy issues. Once we have a list of review candidates, we scour the web for reliable reviews and news reports to see if there have been any breaches, a history of owner complaints, bugs, or other security problems that may disqualify a device from becoming a potential pick.

If something about a product looks questionable, there’s a good chance we won’t even waste time reviewing it. If it still looks interesting or readers are asking about it, we’ll research those reported incidents to see if there’s a reason to reconsider.

We read privacy policies

Most people just click the Yes button repeatedly when installing a new app. We actually read the app’s privacy policy first—all of it. In fairness, if everyone were to read the privacy policies for all the devices and services they use, they would probably never buy or download anything. So we specifically look for red flags: policies or practices that are out of the ordinary for a particular category of device.

For instance, many people decried Ring when it was revealed that the company was potentially sharing Neighbors content with law enforcement agencies around the country and had incorporated a feature in its Neighbors app to make it easy for law enforcement to directly solicit user videos (Ring has eliminated that capability). Our research showed that the practice was more common across the industry than most people realized, with Arlo, Wyze, and Google Nest, among others, acknowledging in their respective privacy policies that they may comply with subpoenas or other formal requests from law enforcement. Although some of these things may be common for a particular product category, others may not.

We work with outside sources

We’re continuing to look for ways to test for and evaluate security and privacy issues. That includes bringing in outside experts to run penetration tests. For our indoor security camera guide, for example, we asked Bill McKinley, head of information security at The New York Times, to hack-test our top four picks. When it came time to do our smart bulb guide, we sent pick contenders to YourThings, which performed a complete analysis of each bulb’s software, hardware, cloud, and network components. We also monitor several of the picks from our guides with the Firewalla Blue, a device that tracks the communications of all devices on a network and reports which ones are sending out data and where it goes.

If we find any security or privacy issues during our testing, we have an internal meeting to talk about what it means, how many owners it could affect, and whether these findings should alter our recommendations (see below for more on this topic).

We ask questions

We put each potential smart-home pick through an extensive and ongoing testing process, but we also vet the company that makes it. Before we decide on our picks, we send a security and privacy questionnaire to each of the relevant companies, asking about what data they collect, how they handle and store it, who has access to it, and much more. For instance, is a lighting manufacturer selling your information to third-party companies? Does a smart plug app include additional security measures such as two-factor authentication? Do security camera companies encrypt your personal data and video transmissions?

Should a security or privacy issue surface, we’ve developed a process to investigate it, called the Security/Privacy Incident Response Update Protocol (affectionately known as SIRUP). Once we learn of an issue, we analyze the particulars, look at how widely people may potentially be affected and how severe the impact may be, ascertain what (if any) remedy exists, and crucially, determine whether the affected company has responded.

Ultimately, we are forced to take companies at their word; however, we think they realize that being dishonest has consequences. Specifically, if we find that a company is acting deceptively or simply responds negligently or otherwise poorly to a security or privacy incident, we’ll make a judgment call on whether the company’s offerings can continue to be picks—or whether we’ll consider them in the future, as well.

We keep testing

Everyone on the Wirecutter staff long-term tests our picks (and not just for smart-home gear—our long-term testing includes everything from robot vacuums to sheets to pet beds). We want to make sure our picks last beyond the original testing period. This is especially important for smart-home devices, where a firmware update, a new app, or changes to privacy policies could completely alter our assessment of them. We also keep track of any problems that may crop up over time, such as security vulnerabilities or whether an item has been discontinued or recalled.

As noted above, should we learn of a potential vulnerability, we follow our Security/Privacy Incident Response Update Protocol to determine what the impact may be and what steps we—and, if necessary, our readers—should take. Our findings sometimes garner wider attention, too: For instance, after we published a report about a problem with Google Nest cameras last year, the company pushed out a fix within hours.

And we rely heavily on reader feedback. We welcome your comments, emails, and social media posts pointing out any flaws you may encounter—but also highlighting any interesting new things for us to test and report on.

It’s Smart-Home Week at Wirecutter! Read more about all the ways your home can become more intelligent.