From a legacy ERP to an engineering platform.
Four-plus years modernizing the production platform of a multi-million-dollar specialty-travel operator: from PHP 5 / CentOS 7 / MySQL 5 to PHP 8.4 / Rocky 9 / MySQL 8.4. Today I lead a two-person engineering team and remain hands-on with the reservations, payments, Linux fleet, and agent infrastructure - including the previews and deployment controls that let other people contribute.
System overview
How the pieces connect. (Architecture shown at a deliberately high level; hostnames, addresses, and vendor specifics omitted.)
The evidence boundary
Production workflows, shared sandboxes, and a factory still under construction.
A Manager Sandbox is a role-specific Tourbot instance running in a Docker container. A branch preview gives a feature its own URL and code checkout, with a shared non-production database. Neither provides per-user data isolation.
ETA Factory is a separate Rust system under construction on live infrastructure, with agent attempts isolated in short-lived Firecracker microVMs and draft pull requests held for human approval. It is not the production deployment path. The separate Kubernetes Demo is a single-node k3s learning and portfolio environment outside ETA production.
The arc
Inherited legacy, modernized in production, then extended.
The business runs on Tourbot - a full-lifecycle group-travel ERP handling the sales pipeline, itinerary building, reservations, passenger logistics, payments, communications, and 70-plus reports, plus three customer- and contractor-facing portals on the same core. When I took ownership it was a PHP 5.6 / CentOS 7 / MySQL 5.7 system whose support runway was running out, and every other part of the business depended on it staying up.
Over four years I led the modernization in production, without a feature freeze: a codebase and fleet migration to PHP 8.4, MySQL 8.4, and Rocky Linux 9 while the business kept running. That baseline supported Docker-based Manager Sandboxes, a Python operations console, fleet-wide metrics and logs, and per-branch previews. PHPUnit now runs on PHP 8.4; CI also checks deployment shell behavior and migration placement.
The work now serves a two-person engineering team and non-engineering managers. A manager can draft a workflow improvement, a stakeholder can inspect a stable preview, and engineering can review the exact production deployment, its schema changes, and its outcome. The platform includes the controls around the application as well as the application itself.
Not every experiment became a lasting service. ProspectForge, the contact-sourcing application described in the original case study, was retired in September 2026. It is historical product work, rather than a current component of the architecture above; its source is archived on GitHub.
By the numbers
Measured outcomes, with their scope kept explicit.
The guide-usage figures are retained from the original June 2026 case study, not presented as current-season totals. Performance figures describe the workloads measured below. The marketing figure counts merged pull requests, not commits, distinct features, or formal review approvals; the public lower bound remains 30+ (32 matched the marketing account and production branch on October 15).
Engineering highlights
Representative work, each problem → approach → outcome.
Manifest performance: 2,650 → 183 SQL statements/request
Problem. The group-manifest page - printed before every trip - took 5-7s and fired ~2,650 SQL statements per render from per-row constructor fan-out. Approach. Six profile-driven rounds: batched in-request prefetch caches, one shared DB handle, and a targeted history-table column + composite index - each verified byte-identical against the original HTML. Outcome. ~1s loads, statements per request down ~93%, validated on a 244-sub-reservation outlier group.PCI-conscious payments
Problem. Aging card integrations and bank-file workflows needed modernizing while customers kept paying daily. Approach. Rebuilt the card flow across merchant processors with bank-account encryption, and modernized the ACH/NACHA bank-file pipeline over SFTP - kept steady through the platform migration. Outcome. Continuity for in-flight registrations and payments through the runtime and database migrations.Multi-tenant AI assistants for managers
Problem. Non-technical managers wanted to query business data and prototype changes safely, without waiting on developer time. Approach. Manager Sandboxes - role-specific Tourbot instances in separate Docker containers behind Traefik, with per-container resource limits and persistent coding-agent sessions. The current design uses one shared writable sandbox database, refreshed daily from production, with per-manager credentials. Pull requests and engineering review control code promotion. Outcome. Managers explore data and draft workflow changes through conversation; the marketing workflow alone has produced 30+ pull requests merged by engineering. Containers separate application files and processes, while shared sandbox data makes refreshes cheaper and allows one manager's database experiments to affect the others.Branch previews for implementation and sign-off
Problem. A shared development host makes long-lived features compete for one environment; stakeholders need to see the code they are being asked to approve. Approach. A pool of up to 30 per-branch Docker previews, with frozen images pinned to a commit for review or editable checkouts for iteration. Outbound integrations are forced into test mode or disabled. Previews share a point-in-time dataset; schema migrations follow production, while row data stays frozen. Outcome. Features can be demonstrated and worked on side by side. Selected editable previews can track their branch automatically, but refuse to overwrite uncommitted work or divergent local commits. The trade-off is shared data: a database experiment in one preview can affect the others.Deployments that survive a closed browser or a restarted worker
Problem. A deployment must ship the commit the operator reviewed, and a lost connection must not leave engineering guessing whether migrations or code reached production. Approach. The review page binds confirmation to an exact commit. A separate worker records the operation durably, serializes production work, checks CI and live state, applies pending migrations and verifies their ledger before the code checkout, and verifies health afterward. Duplicate delivery of the same confirmation creates one operation. Outcome. Closing the browser stops nothing; the operation remains visible in deployment history. After a restart, the worker reconciles its record with live production before finalizing or spending one bounded retry. Uncertain state is held for recovery; code rollback does not promise to reverse database effects. Rollout boundary. Automatic deployment intake from GitHub Actions is implemented but remains disabled pending cutover. The current production workflow is operator-confirmed.Observability first, then a measured 80% database win
Problem. The bespoke status prober answered "is it up?" but not "why is database CPU at 65%?" - no metrics history, paging logic embedded in a dashboard, and performance work that was anecdotal rather than measured. Approach. Stood up Prometheus, Grafana, and Alertmanager on a Terraform-provisioned droplet - node, mysqld, and blackbox exporters across the initial 14-host rollout (host metrics, replication health, uptime, TLS expiry), with the rule-to-SMS paging chain verified live before cutting over the old pager. Then usedperformance_schema to rank 27 days of production queries by total time: three full-table-scan hot queries dominated, fixed with three composite indexes applied as online DDL (INPLACE, LOCK=NONE), EXPLAIN-verified, rollback written first.
Outcome. One paging pipeline, capacity questions answered from dashboards, and ~80% of measured query time (~404K of ~500K query-seconds) gone - with the remaining load diagnosed as application-level N+1 volume rather than papered over with hardware.
Platform migration, no downtime
Problem. PHP 5.6, MySQL 5.7, and CentOS 7 were all past or near end-of-life under a live business. Approach. Codebase-wide PHP 5→8 migration, a MySQL 5.7→8.4 plan with dual-running databases during cutover, and a CentOS 7→Rocky 9 fleet migration with per-server runbooks. Outcome. A current, supported stack across the fleet - completed without an outage staff or customers noticed.Stack & why
The reasoning, not just the list.
Open source: I upstreamed the MySQL 8.4 caching_sha2_password auth-handshake fix this stack needed into php-mysql-replication (PR #148).
Architecture decisions: the non-obvious trade-offs - tenant isolation, the data tier, the deploy pipeline, agent identity - are written up as sanitized ADRs at adr.stephens.page.
Verified October 15, 2026 against the current repository documentation, CI configuration, and read-only checks of the deployed operations console and deployment worker. Some capabilities in the repositories are still awaiting rollout; those are identified above.
Confidentiality: employer repositories remain private. This account describes architecture at a high level and omits private endpoints, network addresses, credentials, and customer data.
Read further
The reasoning, the résumé, and the rest of the work.