Table of Contents

Architecture

Architecture Diagram

1. Data Intake

New items can be added however desired. A pre-build web API is included, as well as a queue listener which allows for asynchronous, scalable processing.

2. Rules

This is the primary workitem processing engine

The processing engine is responsible for creating Jobs for dispatch. The dispatching engine (detailed below) is responsible for determining how to transmit the job.

The item processing engine is currently built around a rule engine (NRules), which allows for data-centric routing, though also allows for more standard step-by-step workflow processing. Several support classes allow for easy writing of the routing rules.

3. Job Dispatch and Transport

Sends the job to workers.

This includes queuing jobs in Storage Queues, Service Bus Queues, Redis, or potentially any other transport. Also, Jobs need not be immediately dispatched by the system - for example, instead of directly queueing the job for a background worker, the job could be advanced through the rules by waiting for the user to click a button on a UI.

Typically, Jobs are dispatched as well-defined JSON messages, allowing workers to process the message without requiring the use of an SDK.

4. Job Scheduling

Schedules jobs for future processing.

Currently Hangfire and scheduled Service Bus Queue messages are supported. Using the schedule Service Bus messages provides a lower-cost solution by allowing auto-shutdown of the router and/or result processing if it is running as a separate Function – once the message is enqueued by Azure, the system can auto-start from a trigger. Hangfire, on the other hand, requires an always-on system while providing more visibility and configurability. Hangfire is also a good choice for on-premise solutions.

5. Result Processing

Processes results sent back from workers.

This can run as a background processor within an ASP.Net application, or as a separate Function which would allow for greater scalability.

Results are sent as standard Messages, which means workers aren't required to rely on an SDK to communicate with the server. However, the provided libraries can be used to make things easier.

6. Document Storage

Stores "documents," which includes the primary work item data as well as attachments.

Currently uses Blob Storage, though Cosmos is being implemented which would allow for high throughput and query abilities.

7. System Storage

Data and structures the support core system processing. Tracks job statues and schedules.

  • SQL – Primary job structure storage. Utilizes Entity Framework Core.
  • Caching - Uses ASP's IDistributedCache.