Why I Moved Development to VMs
Motivation
I've had it with supply chain attacks.
The recent inclusion of malware into the nx package was the last straw for me. Malware being distributed in hijacked packages isn't a new phenomenon, but this was an attack specifically targeting developers. It publicly dumped user secrets to GitHub and exposed private GitHub repos publicly.
I would have been a victim of this malware if I had not gotten lucky. I develop personal projects in Typescript. I've used nx. Sensitive credentials are stored in my environment variables and configs. Personal documents live in my home directory. And I run untrusted code in that same environment, giving any malware full access to all my data.
How did this happen (again)?
First, the attackers utilized a misconfigured GitHub Action in the nx repo using a common attack pattern, the pull_request_target trigger. The target repo's GITHUB_TOKEN is available to the source repo's code in the pull request when using this trigger, which in the wrong case can be used to read and exfiltrate secrets, just as it was in this incident.
💭 This trigger type is currently insecure by default. The GitHub documentation contains a warning about properly configuring permissions before using
pull_request_target, but when security rests on developers reading a warning in your docs, you probably have a design flaw that documentation won't fix.
Second, they leveraged script injection. The workflow in question interpolated the PR title directly in a script step without parsing or validating the input beforehand. A malicious PR triggered an inline execution of a modified script that sent a sensitive NPM token to the attacker.
💭 Combining shell scripts with templating is a GitHub Action feature that is insecure by design. There is a reason why the GitHub documentation is full of warnings about script injection. A more secure system would require explicit eval of all inputs instead of direct interpolation of inputs into code.
My New Setup
I'm moving to development in VMs to provide stronger isolation between my development environments and my host machine. Lima has become my tool of choice for creating and managing these virtual machines. It comes with a clean CLI as its primary interface, and a simple YAML based configuration file that can be used to customize each VM instance.
Why Lima, not Vagrant or Containers?
Despite having many years of experience using Vagrant and containers, I chose Lima instead.
From a security perspective, the way Vagrant boxes are created and distributed is a problem for me. The provenance of these images is not clear once they're uploaded to Vagrant Cloud. To prove my point, I created and now own the linuxmint and manjaro Vagrant registries. To my knowledge, there's no way to verify the true ownership of any registries in Vagrant Cloud.
Lima directly uses the cloud images published by each Linux distribution. Here's a snippet of the Fedora 42 template.
images:
- location: https://download.fedoraproject.org/pub/fedora/linux/releases/42/Cloud/x86_64/images/Fedora-Cloud-Base-Generic-42-1.1.x86_64.qcow2
arch: x86_64
digest: sha256:e401a4db2e5e04d1967b6729774faa96da629bcf3ba90b67d8d9cce9906bec0f
Not perfect, but more trustworthy.
I also considered Devcontainers, but I prefer the VM solution for a few reasons.
While containers are great for consistent team environments or application deploys, I like the stronger isolation boundary that VMs provide. Container escapes and kernel exploits are a class of vulnerability that VMs can mitigate and containers do not. Finally, the Devcontainer spec introduces complexity I don't want to manage for personal project development. I want to treat my dev environment like a persistent desktop where I can install tools without editing Dockerfiles. VMs are better suited to emulate a real workstation without the workarounds required by containers.
How I Configure Lima
Out of the box, most Lima templates are not locked down, but Lima lets you clone and configure any template before creating or starting a VM. By default, Lima VMs enable read-only file-sharing between the host user's home directory and the VM, which exposes sensitive information to the VM. I configure each VM with project specific file-sharing and no automatic port forwarding.
Here's my configuration for rustlings.
minimumLimaVersion: 1.1.0
images:
- location: https://download.fedoraproject.org/pub/fedora/linux/releases/42/Cloud/x86_64/images/Fedora-Cloud-Base-Generic-42-1.1.x86_64.qcow2
arch: x86_64
digest: sha256:e401a4db2e5e04d1967b6729774faa96da629bcf3ba90b67d8d9cce9906bec0f
# Only include a project folder specific mount
mounts:
- location: "~/projects/rustlings"
mountPoint: /var/home/cruatta/projects/rustlings
writable: true
# Disable automatic port forwarding. Any ad-hoc port forwarding can be done via SSH
portForwards:
- ignore: true
proto: any
guestIP: "0.0.0.0"
# Force a specific SSH port for IDE integration
ssh:
localPort: 36475
# Explicitly specify resource allocations
cpus: 4
memory: "4GiB"
disk: "25GiB"
This template can then be used to create a VM instance
⬢ [fw13] ~ ❯ limactl create --name=rustlings rustlings.yaml
After creation of the VM is complete, accessing it over SSH can be done transparently via the shell subcommand.
⬢ [fw13] ~ ❯ limactl shell rustlings
[cruatta@lima-rustlings cruatta]$
The VM is now ready to be connected to my IDE.
Connecting my IDE
I'm mostly a JetBrains IDE user. These IDEs have a Remote Development feature that enables a near local development experience with VMs. A client-server communication model over an SSH tunnel enables this to work.
Connecting my IDE to my VM was a 5 minute process that included selecting my Lima SSH config (~/.lima/<vm>/ssh.config) for the connection and picking a project directory. The most time consuming part of this was waiting for the IDE to download the server component to the VM. After that, the IDE setup was done. I had a fully working IDE and shell access to the VM in the IDE terminals. I haven't found any features that don't work as expected.
There is also granular control over SSH port-forwarding between the VM (Remote) and host (local) built in, which is convenient for me when I'm developing a backend application.
The integration between Podman/Docker and these IDEs extends to the Remote Development feature as well. I can run a full instance of Podman within my VM, and once the IDE is connected to the VM's instance of Podman, I can easily forward listening ports from my containers back to my host.
Wrap up
The switch to VMs took me an afternoon to set up and I get the same development experience with actual security boundaries between untrusted code and my personal data. Lima has made VM-based development surprisingly painless and I'm worried a lot less about the next supply chain attack.