A good way to secure an app is to (1) start with the assumption that an attacker will get root at some point and make their life hell accordingly, and then (2) make sure that an attacker doesn’t get root. Sometimes you hear the platitude “if they get root, it’s over anyway” used as a justification for skipping steps. Okay, yes, but why make it easier for them if hampering them doesn’t make the deployment much harder for you?

This weekend, I created a small Spring Boot backend service in order to feed select financial data to a simple Android dashboard app I’d built for myself and my wife. The service used a personal access API token from a financial institution in order to get our data, so I wanted to make sure the public internet exposed service would be secure. As an additional constraint, I also wanted to add a minimal amount to my monthly webhosting bill.

In this article, I walk you through my choices and thinking around:

how to choose a VPS provider,
how to secure the backend service from an attacker with root, and
how to secure the backend service against an attacker getting root.

Selecting a VPS Provider

I often use simple VPS providers like Digital Ocean, Linode, or Vultr, because they have straightforward pricing and easy configuration. There are lots of these kinds of providers out there, but I mention these three to start because they’ve been around for a while. I’ve seen better deals, but who knows if besthostingdeals.xyz is going to vanish in two months because their ultra cheap pricing wasn’t sustainable.

Vultr

I started with Vultr, since they have a $2.50/mo IPv6-only hosting option. The IPv6-only part presented a few problems though.

For starters, I would need the Android app to call my backend with TLS so that the authorization credentials would be secure in transit. Since I wanted to use LetsEncrypt for certs, I’d need to tie my IP address to a domain. Unfortunately, my DNS provider’s policy on IPv6 is “call us to set that up”. You can’t configure it through their GUI like you can with IPv4. Not insurmountable, but annoying.

Much more troublesome was that that Vultr’s IPv6 VPS instances can’t connect to non-IPv6-enabled hosts. That means no Docker registry. That was a deal breaker. I could upgrade to one of their IPv4 instances, but then the price advantage would vanish.

Amazon Web Services

Amazon LightSail is AWS’s simple VPS offering, and they have a tiny VPS instance available for only $3.50/mo, so I tried there next. But alas, I ran into problems with LightSail too.

How to (Partially) Protect Credentials Against a Root Attacker

Let’s talk about securing against an attacker with root. My backend service needed to have two sensitive credentials: the financial institution’s personal access token (mentioned above) and the Spring Security user password for the Android app to log in with. How could I deliver those to the backend service without also making it easy for an attacker to get them?

Make them environmental variables in the Docker Compose file? Nope, then our root attacker can see them with docker inspect.
Pass them as parameters in the Docker ENTRYPOINT (or similar)? Nope, then our root attacker can see them with docker ps.
Forget Docker, just make a temporary environmental variable, launch the backend service manually, and then delete the environmental variable? Nope, because ugh. (But more importantly, what happens if the VPS instance reboots? And also, our root attacker may be able to search your console history.)

Seems Like This Should Be Solved Already…

It is. The enterprise term for this problem is “secrets management”, and there are various solutions available to deal with it. Microsoft has Azure Key Vault. Amazon has AWS Secrets Manager. Both are fairly cheap if you use them correctly. Hashicorp offers Hashicorp Vault hosting, but also lets you host Vault yourself for free.

The way these secrets management solutions work is, they store your secrets (sensitive credentials) securely at rest and send them securely in transit on request. It’s not perfect, because in order to get your secrets from the vault, you still have to authenticate somehow, but the way you authenticate is usually better than the alternatives.

Back to Business

Sounds great - I chose Hashicorp and self-hosted, end of story, right? Not quite. Per Hashicorp’s docs, the minimal requirements for Hashicorp Vault are roughly equivalent to an AWS m5.large instance, which would mean, at today’s prices, roughly $70/mo. Ouch. That’s too much for my little dashboard app.

Instead, I went with Amazon’s AWS Secrets Manager. Pricing is pretty similar to Microsoft’s offering: $0.40 per secret per month, plus $0.05 for every 10k calls to the Secrets Manager API to retrieve a secret. (It’s very unlikely we’ll exceed 10k such API calls in a month.)

Enter Jasypt

We can do a little better than that though. Jasypt for Spring Boot is a library which makes it trivial to encrypt any sensitive properties in an application.properties file. It decrypts them with a single master password, reducing the number of secrets in Secrets Manager from n to 1.

Service-Linked IAM Roles

I promised problems.

This Lightsail + Secrets Manager setup looked like it was going to work, but when I tested it, I ran into a snag. Lightsail instances are not like EC2 instances in that you can’t edit their IAM Roles to add permissions. All Lightsail instances have a “service-linked role” called AmazonLightsailInstanceRole, and it doesn’t have the SecretsManagerReadWrite permission required to connect to Secrets Manager.

Thus, I had to switch to an EC2 t2.nano instance, which costs about $1.30/mo more. (Notably, EC2 instances with this permission only get to read, not write, despite the name - this is clarified elsewhere in the AWS GUI.)

It’s worth mentioning here that once the EC2 instance has the SecretsManagerReadWrite permission, you don’t need to add AWS credentials anywhere. As a part of the provisioning process, a link-local address is set up within the EC2 instance which allows the Secrets Manager SDK to communicate with AWS-internal authentication stuff. Neat, eh?

More Hardening

Before moving on, let’s quickly summarize. If an an attacker managed to get root on my EC2 instance, in order to get access to my data, they would have to first decompile my jar and extract application.properties, then either

brute force the decryption of the Jasypt-encrypted personal access token or
authenticate to Secrets Manager to get the Jasypt master password.

That’s quite a bit harder than docker inspect <hash> or printenv. Still, it’s not impossible, so let’s prevent our theoretical attacker from getting root in the first place.

Limiting Access

Though putting up barriers against a root attacker is worthwhile, a determined and competent root attacker will get past everything in time. Thus, you must prevent an attacker from gaining root access to begin with, and the name of that game is limiting access.

EC2 Settings

In the EC2 Security Groups settings, the default open ports were 80, 443, and 22. This is a pretty sensible default, but AWS also lets you restrict traffic per port to an IP address you specify. There’s an option for “My IP”, which is perfect to restrict SSH with.

Running the Jar as a Non-Root User

Per snyk.io’s excellent Docker best practices article, the attack surface can be reduced significantly if you don’t run the application as root inside your Docker container. (This prevents things like container escape via mounted volumes with poorly thought out permissions.)

Et Cetera

There are lots of standard best practices that are broadly applicable with web services. Whole tomes are written on the subject and whole companies exist to scan your infrastructure. The subject is far too large for this article. I mentioned just a few here because I found them helpful. If you want to learn more, the OWASP Top Ten is a good place to start.

Other Considerations

There were other options I decided against implementing, for various reasons.

Why Not Docker Secrets?

I did look at using Docker secrets. Surprisingly, you can use them through Docker Compose, even if you’re not using Docker Swarm. (This directly contradicts the documentation, and there are vehement disagreements on StackOverflow about whether it works.)

The advantage of using Docker secrets, is, as far as I can tell, that you’re not using environmental variables or ENTRYPOINT parameters to hide secrets, which makes them less visible to an attacker. Still, if the attacker looks at your decompiled jar, they’re going to see where those secrets are being read from, which kinda makes Docker secrets not much better than a config file.

What About Bytecode Obfuscation?

I’m less decided about this one. It’s usually said that bytecode obfuscation options (like ProGuard or yGuard) are more useful for client code than server code because from the beginning, you’re putting your source in someone else’s hands. That’s true, but even server-side, it’s yet another obstacle for an attacker. That said, unlike these other mitigations, obfuscation can be somewhat difficult to set up, especially for something like a Spring Boot app where there’s a lot of reflection involved.

In my case, I didn’t bother because I didn’t think the extra effort was commensurate with the likely threat.

Closing Thoughts

So there it is: ports restricted, access authenticated, credentials hidden with secrets management, for about $5.20/mo. Maybe I’ll do more in the future if I add more sensitive things to the dashboard app, but for now, I’m pretty happy with my weekend project.