Logs and diagnostics

There are three places to look: the serial console, the service logs under /system/logs, and the metrics that sysbox stats, ps, top, free, and ss read. For a process that is stuck, mdbg dumps its stacks.

The serial console

The kernel writes its log to the serial port as seconds:millis cpu LEVEL target:line - message, at level Info by default. Once sys-tty has started, kernel output is forwarded through it so that kernel and user output do not interleave. Boot messages, the status of sys-io if it ever exits, panics, and the output of headless services all end up here, which is why the console is the only diagnostic channel when networking or sys-io itself has failed. Keep it: with run-qemu.sh the console is the terminal you started the VM from, so redirect stdout and stderr to a file for long runs, and enable serial console capture on a cloud provider before first boot.

Service logs

Services log through moto-log, a log-crate facade that sends records over IPC to strobe. strobe writes one file per registered tag, /system/logs/TAG.log, and rotates it to TAG.log.prev. The directory is rwxr-x--- and the files rw-r-----: strobe, running as System, alone creates and rotates them; an Interactive session can list and read them; a None-role process cannot enter the directory. Sending a record requires CAP_LOG, which is checked against the sender's kernel-reported capabilities, not against anything the sender claims.

Ordinary programs do not log this way. A panic message, a backtrace, and the runtime's own warnings go to the process's stderr first, so they reach the terminal or the file that captured it; the kernel log is a fallback used only when that write fails and the process holds CAP_LOG. This is what keeps diagnostics visible for an interactive command, which by default has no logging capability.

Metrics

Metrics are federated and discovered at run time, with no hardcoded list. A metric is identified by provider, name, and scope (a PID or global). The kernel's catalog is self-describing and read with a syscall; userspace providers such as sys-io register with the registry that strobe hosts, and a collector lists the registry and then asks each provider to describe and report its metrics. This avoids both a syscall-only interface, which cannot see userspace producers, and a procfs, which would create a cycle since the filesystem is itself a userspace service.

Names worth knowing: the kernel's mem.* admission counters and low-water marks (see Memory); sys-io's net.* family (device packets, bytes, and drops, socket and listener counts, half-open and backlog counts, SYN cookie counters, suppressed RSTs and ICMP errors, reassembly outcomes, pressure counters) and fs.* family (capacity and available bytes, cache hits and misses, read and write counts and latencies, readahead, device I/O, channels).

Stack traces from a running process

There is no source-level debugger. mdbg print-stacks PID (on the developer image, under /devtools/bin) pauses the process through the kernel's debug interface, walks each thread's stack, and prints raw addresses, marking frames that are inside the runtime. Symbolize them on the host with addr2line against the unstripped binary from the build tree, for example build/obj/sys-io/x86_64-unknown-motor/debug/sys-io, and against build/obj/vdso/x86_64-unknown-motor/debug/rt for runtime frames. This is the first thing to reach for when a process hangs: it distinguishes a lost wakeup from a deadlock in a minute. When a program cannot be reached over SSH, drive mdbg from the serial console.

When sys-io stops

sys-io owns live filesystem and network state that cannot be rebuilt safely in a replacement, so if it exits or is killed the kernel logs its status on the console and halts the VM. Do not expect the machine to recover on its own; read the console.