Lots of folks are moving to X.509 certificate-based authentication as they start to use the Azure IoT Hub Device Provisioning Service, which is great! But I've gotten lots of questions about what the best practices are, and how to go about doing it at scale. There are a lot of variables that influence where certs come from and how they are installed, and who owns each stage depends on the specific processes and business relationships in place. This blog post is meant to provide some clarity around the cert generation and installation process for IoT devices at production-level scales.
One note before I begin, if you already have a system in place for installing certificates on your IoT devices and its working out for you, great! Feel free to stop reading and check out some of our other content. This blog post is for folks who are just making the switch to using certificates on IoT devices and are struggling with figuring out what works best.
Mandatory security rant
If you're coming from the land of passwords and want a quick fix, you might be asking yourself why you can't use the same certificate in all your devices like you would be able to use the same password in all your devices. First, using the same password everywhere is really bad practice and has already caused several major DDoS attacks, including the one that took down DNS on the US East Coast a few years back. So don't ever do this, even with your own personal accounts. Second, certificates are not passwords they are complete identities. A password is like knowing the secret phrase to get into the clubhouse, a certificate is the bouncer asking to see your driver's license. It’s like if I got multiple copies made of my passport and gave you one to use for identification. Either you give my name and my passport and impersonate me (AKA spoof my identity), or you give your name and my passport and the border control agent kicks you out for not having an ID matching your name.
Variables involved in certificate decisions
There are several variables that come into play when you decide to use certificates in your IoT devices. The heavy hitters are as follows.
Where the certificate root of trust comes from. Make this decision based on your own cost tolerances and masochism levels – managing a public key infrastructure (PKI), especially if your company does not have experience doing so, is not for the faint of heart. Your options are:
- 3rd party PKI in which you buy intermediate signing certificates (or a private CA) from a 3rd party cert vendor.
- Self-managed PKI in which you maintain your own PKI system and generate your own certificates.
- Azure Sphere’s security service (only for Azure Sphere devices).
Where certificates are stored. This depends on the type of device you're building, the expected device margins (whether you can afford secure storage), the device's capabilities, and existing security technology on the device that may be leveraged. Your options are:
- In a hardware security module (HSM) (recommended!). Check whether your device's control board already has an HSM installed. I've talked to a couple customers who were surprised to find they had an HSM already installed in their devices that they just weren't using. If you know you don't have an HSM, work with your hardware manufacturer to identify an HSM that meets your needs.
- In a secure place on disk such as a trusted execution environment (TEE).
- Local file system or cert store, e.g. the Windows certificate store.
- Other
Connectivity at the factory. This determines how and when you get the certificates that will be installed on the devices. Your options are:
- Yes connectivity (can generate certs locally).
- No connectivity (factory has a signed CA cert it uses to generate device certs locally offline).
- No connectivity (certs need to be generated ahead of time, OR offline PKI).
Audit requirement. Depending on the type of devices you are producing, you may be required by regulations to have an auditable trail of how device identities are installed in your devices. Only do this if you need to since it can add significant costs to production. If you're not sure if this applies to you, check with your company's legal department. Your options are:
- Not a sensitive industry, no auditing required.
- Sensitive industry, certs need to be installed in a secure room according to various compliance certification requirements. If you need a secure room to install certificates, you probably already are aware of how certs get installed in your devices and already have a system in place. This post is probably not for you.
Length of cert validity. Like a driver's license, certificates have an expiration date that is set when they are created. Your options are:
- Very long, never will need to roll it. A very risky approach. Risk can be reduced using very secure storage like an HSM etc. However, the long-lived certificate approach is not recommended.
- Will need to roll during the lifetime of the device. This is context dependent and requires a strategy for how to roll certificates, where you're getting certificates from, and what type of over the air functionality exists for your devices.
When to generate certificates
As I mentioned in the previous section, connectivity at the factory plays an important role in determining when you generate the certificates for your devices. Let's talk about that.
Some HSM vendors offer a premium service in which the HSM vendor installs certificates in the HSMs on behalf of the customer. Customers give the HSM vendor access to a signing certificate and the HSM vendor installs certificates signed by that signing certificate on each HSM the customer buys. All the customer has to do is install the HSM on the device. One customer I've talked to who went this route says the process is pretty slick. If you choose to go the “preloaded in HSM” route, when to generate certificates is not your problem! Hooray!
If your devices internally generate their certificates, then you must extract the public X.509 certificate from the device for enrollment.
If your factory has connectivity, you can generate the certs whenever you need them.
If your factory does not have connectivity, and you are using your own PKI with offline support, then you can generate the certs whenever you need them.
If your factory does not have connectivity, and you are using a 3rd party PKI, you have to generate the certs ahead of time from an internet connected location.
When to install certificates
Now that you've generated certificates for your IoT devices (or know when to generate them), it's time to install them into your device.
If you go the “preloaded in HSM” route, your life is made a little easier. Once the HSM is installed in the device, it is available for the device code to access. You will then use the HSM APIs to use the cert stored within the HSM. It's the easiest option, but it costs a little more.
Otherwise, you need to install the certificate as part of the production process. The easiest way to integrate it into your existing process is to install the certificate with the initial firmware image. Each device has to go through at least one touch on the manufacturing floor to get an image installed and to run the final quality checks, etc, before the device is packaged up to send to its final destination. There's generally a tool used that does the installation and testing in one shot. You can modify the tool to generate a cert or pull a cert from a pre-generated store and then install it wherever the cert needs to be installed. The tool gives you the scale you need for production-level manufacturing.
If you need more help getting certificates in your IoT devices, please reach out to someone from our security auditor program. Once you get set up with certs on your devices, learn how to use certificates and enrollments in the Device Provisioning Service!
To sum things up with a limerick:
Installing device certs at scale
Can be fraught with pain and with fail
You can totally do it!
Just plan and work through it
This blog post serves as your guardrail.