Distributed automation · Architecture case study
From scheduled job to isolated execution
An automation platform must do more than start work. It needs to track what happened, handle interrupted execution, and keep every operation within the right tenant boundary.
- Project context
- Multi-tenant enterprise automation
- My role
- Solution Architect & Principal .NET Engineer
- Focus
- Orchestration, resilience & workload isolation
01 / The challenge
Starting a job is the easy part.
The platform needed reusable workflows, immediate and scheduled execution, and a way to track a requested run separately from the workload carrying it out. Jobs also had to operate within tenant, organization, and project boundaries.
I helped define an architecture that separated those responsibilities into independently deployable services, with APIs for direct interactions and events for asynchronous coordination.
02 / Platform architecture
Clear ownership.
Coordinated execution.
Domain services own business behavior. Events connect their lifecycles, while Kubernetes hosts the services and isolated workloads.
Flow & job
Reusable workflows and executable definitions
Schedule
Immediate, one-time and recurring triggers
Run
The requested operation and its overall state
Execution
Workload creation and execution status
Tenant & organization
Customer context and organizational boundaries
Project
Project scope within the tenant
User & access control
Identity, roles and permissions
Notification & environment
Status communication and protected configuration
03 / Execution lifecycle
Follow one scheduled job.
The run records the requested operation. The execution records the workload and its state. Events carry the changes between services.
Define the job
Save the execution details and optional schedule.
Publish the change
The job service emits a creation or update event.
Register the schedule
The schedule service consumes the event and records its timing.
Create a run
A trigger creates the record of the requested operation.
Create an execution
Record the execution and publish the workload request.
Launch the workload
The execution service creates an isolated Kubernetes workload.
Track its state
Use explicit states: Created, Started, Succeeded or Failed.
Publish completion
Update the run, notify interested services, and retain history.
The platform supports both scheduled and immediate execution. The sequence above follows the scheduled path.
04 / Architectural reasoning
Design for the second delivery.
Retries, restarts, and duplicate messages were part of the reliability design. Each choice also creates an operating responsibility.
Give each domain a clear owner.
Workflow definition, scheduling, run tracking, and execution have separate responsibilities. Domain-driven design, CQRS, and explicit contracts organize the services around those responsibilities.
Tradeoff Independent deployment adds distributed coordination and operational work. The boundaries need to justify that cost.
Make repeated delivery safe to handle.
The reliability patterns include idempotent consumers, duplicate protection, retry-safe commands, and explicit status transitions.
Engineering consideration A retry policy alone is insufficient. Each command needs a defined response when the same request has already changed state.
Carry enough context to investigate a failure.
Correlation IDs, structured logs, failure events, and run and execution histories connect the work across service boundaries.
Engineering consideration An execution failure and a delayed status update need to remain distinguishable when investigating what happened.
Let event contracts evolve deliberately.
Event versioning is part of the documented reliability approach, alongside explicit producer and consumer responsibilities.
Tradeoff Services can ship independently only when changes account for the consumers that still depend on an earlier contract.
05 / Security & operations
Tenant context travels
with the work.
The design validates context at the request, domain operation, persistence, event-handling, and workload-execution boundaries.
Identity & authority
- Token-based authentication
- Policy-based authorization
- User and service identity
- Roles and permissions
Tenant boundaries
- Tenant, organization, and project scope
- Commands, queries, and domain entities
- Events and data access
- Workloads, logs, and audit context
Workload operations
- Dynamic Kubernetes workload creation
- Health and readiness checks
- Declarative configuration and rolling updates
- Argo CD delivery and protected configuration
When a schedule triggers work, the execution path still needs the authorized tenant and project context. The original browser session cannot be the only place that relationship is checked.
Isolation depends on application controls as well as workload boundaries.06 / My contribution
Architecture with
hands-on delivery.
My contribution covered domain and service boundaries, event contracts, multi-tenant security, scheduling and execution lifecycles, and Kubernetes workload design.
Architecture
Service & lifecycle design
Defined boundaries, API and integration patterns, datastore responsibilities, resilience patterns, and the relationship between schedules, runs, and executions.
Implementation
.NET services & messaging
Developed services, commands, queries, and validators. Designed Kafka producers and consumers, implemented MongoDB persistence and revisions, integrated scheduling, and implemented policy-based authorization.
Operational engineering
Failure diagnosis
Diagnosed messaging and distributed-system failures, supported production troubleshooting, and contributed architecture decisions and code reviews.
.NET 9 · C# · ASP.NET Core · FastEndpoints · MediatR · FluentValidation · Apache Kafka · MongoDB · PostgreSQL · Redis · Docker · Kubernetes · Argo CD
Delivered capabilities include independently deployable services, immediate and scheduled execution, traceable run histories, and isolated workloads. Measured customer outcomes and performance benchmarks are outside the scope of this study.