BrokenOps

BrokenOps makes infrastructure troubleshooting the product. A learner launches an intentionally faulty Linux machine, diagnoses it in a browser terminal, repairs the service, and asks the platform to verify the result.

That requires more than a frontend around shell commands. The implementation connects guest provisioning, virtual disks, network access, SSH sessions, verification scripts, and lab cleanup. Its supported environment is a native Linux host with KVM/libvirt, rather than a publicly hosted lab service.

From a lab definition to a VM

The FastAPI backend reads a lab’s YAML definition and creates a qcow2 overlay backed by an Ubuntu 24.04 image. Cloud-init receives the lab setup and an injected SSH public key. The backend builds a cidata ISO with xorriso, then asks libvirt to define and start the guest.

The virtualization engine connects to qemu:///system, supplies the requested CPU, memory, disk, and virtual network, and uses a NAT libvirt network. It can fall back to QEMU emulation if KVM is absent, but that is not evidence of equivalent performance or a supported nested-virtualization setup.

Terminal and service access

The React frontend is served by Nginx. API and WebSocket requests reach FastAPI, which establishes AsyncSSH connections to the guest. This lets the browser terminal interact with the machine without requiring the learner to configure a local SSH client.

The platform also proxies exposed service ports, restricted to the ports listed in the lab definition. Readiness is a separate operation: the status endpoint waits for an IP address and for cloud-init to finish before writing a ready marker. A background task checks those markers and stops labs after roughly an hour from that marker.

A concrete Nginx exercise

The Nginx verifier illustrates the model. Provisioning installs Nginx but changes its listener to the invalid port 80808. A learner needs to inspect the service and its configuration, repair the listener, and return Nginx to a running state on port 80.

Verification runs the lab’s shell script over SSH. A zero exit status maps to a successful result. Checking both service activity and the listening port makes this more meaningful than simply asking whether a configuration file changed.

Checking the labs in CI

The lab-change workflow installs KVM/libvirt on an Ubuntu runner, prepares keys and the base image, starts Compose, and invokes the lab test runner. That script launches changed labs, applies their solution.sh, runs verification, and checks exposed ports.

This tests the exercise lifecycle rather than just the syntax of the YAML. The repository has a recorded successful lab-verification run from June 2026, but that run predates the reviewed snapshot. It is historical execution evidence, not a claim that every current path has been revalidated.

The host is part of the system

The backend container has privileged access, host networking, /dev/kvm, and the libvirt socket. Host setup also relaxes several firewall and virtualization security settings. Those are substantial local-lab assumptions; this is not an isolated web container suitable for dropping onto a shared production host.

Source review also exposes a restricted-terminal branch that references an unassigned variable. That path needs correction before restricted-mode support can be treated as dependable. Shell functions that discourage guest shutdown are convenience protection, not a security boundary. The strongest part of the project is the reproducible exercise lifecycle, with clear room to strengthen host isolation and access handling.

Source files

• VM engineDisks, libvirt domains, and virtual networking.
• Backend APIReadiness, terminal access, verification, and cleanup.
• Runtime configurationHost integration and container privileges.
• Lab verification runnerAutomated launch, solve, verify, and port checks.