Asynchronous Document Processing and Analytics API

This project is an enterprise-grade, asynchronous document processing system built on Azure. Instead of a standard synchronous CRUD application, it implements an event-driven architecture where users upload files, the system processes them in the background using AI, and users can securely query the results.

Architecture

This project is built using a microservices-inspired, serverless architecture.

1. Gateway & Authentication (Azure Security & APIM)

• Microsoft Entra ID: Secures the API. Only authenticated users with valid JWT tokens can make HTTP requests.
• Azure API Management (APIM): Acts as the front door, applying rate-limiting policies and JWT validation before routing requests to the backend.

2. Core Compute (Azure Compute Solutions)

• Azure Functions (Isolated Worker Model): Hosts both the core REST API (handling immediate user requests) and the serverless background processors.
• Polly: Implements resilience patterns (Retries and Circuit Breakers) for outbound HTTP requests to Cognitive Services.

3. Storage & Databases (Azure Storage)

• Azure Blob Storage: Stores the raw uploaded documents. The API generates short-lived Shared Access Signature (SAS) tokens for secure upload and download operations.
• Azure Cosmos DB (NoSQL): Stores document metadata, processing status, and the extracted analytics results.

4. Messaging & Event-Driven Routing (Connect and Consume Services)

• Azure Event Grid: Listens for Microsoft.Storage.BlobCreated events. When a file is uploaded, it triggers the first background function.
• Azure Service Bus: Provides reliable message queuing. After initial event validation, a message is pushed to a Service Bus queue to be picked up by the AI processing function.

5. Security & Secrets (Implement Azure Security)

• Managed Identities: Uses System-Assigned Managed Identities (DefaultAzureCredential) to authenticate between Azure resources (Functions, Service Bus, Cosmos DB, Blob Storage, Cognitive Services) without hardcoded credentials.
• Azure Key Vault: Safely stores any legacy connection strings or external API keys required by the application.

6. Optimization & Monitoring (Monitor, Troubleshoot, Optimize)

• Azure Cache for Redis: (Planned) Caches document metadata to optimize read speeds and reduce Cosmos DB Request Units (RUs).
• Application Insights: Tracks distributed telemetry, exceptions, and execution times across the asynchronous pipeline.

System Workflows

1. Secure Upload Flow

1. Client calls POST /api/documents with file metadata.
2. The API creates a DocumentModel record in Cosmos DB with a status of UploadPending.
3. The API generates a write-only Blob SAS URL and returns it to the client.
4. The client uploads the file directly to Azure Blob Storage using the provided SAS URL.

2. Asynchronous Processing Flow

1. Blob Storage fires an event to Event Grid.
2. StatusChangeTrigger intercepts the event, updates the Cosmos DB status to Processing, and sends a payload to Service Bus.
3. DocumentProcessingTrigger picks up the Service Bus message, generates a read-only SAS URL, and passes it to Azure Document Intelligence via REST.
4. The system polls for completion, extracts the text/analytics, and updates the final results in Cosmos DB.

API Endpoints

The primary HTTP operations are handled by the DocumentOperations function class.

Method	Endpoint	Description
POST	/api/documents	Registers a new document and returns a secure upload SAS URL.
GET	/api/documents	Retrieves all document records for the authenticated user.
GET	/api/documents/{id}	Retrieves metadata and processing results for a specific document.
GET	/api/documents/{id}/download	Generates a secure, time-limited download SAS URL for the processed file.
DELETE	/api/documents/{id}	Deletes the document metadata from Cosmos DB and the physical blob from Storage.

Project Structure

• DocumentProcessor.Api.Endpoints: Contains the HTTP-triggered Azure Functions serving as the REST API (DocumentOperations.cs).
• DocumentProcessor.Api.Triggers: Contains the event-driven Azure Functions (StatusChangeTrigger.cs, DocumentProcessingTrigger.cs).
• DocumentProcessor.Core.Models: Shared data models representing the Cosmos DB schema.
• DocumentProcessor.Core.Enums: Shared enumerations for Dependency Injection keys and Status types.

Local Development Setup

Prerequisites

• .NET 8 SDK
• Azure Functions Core Tools v4
• An active Azure Subscription
• Azure CLI (logged in via az login)

Configuration

Create a local.settings.json file in the root of the API project with the following structure:

"IsEncrypted": false,
"Values": {
"AzureWebJobsStorage": "UseDevelopmentStorage=true",
"FUNCTIONS_WORKER_RUNTIME": "dotnet-isolated",
"StorageAccount__blobServiceUri": "https://<your-storage-account>.blob.core.windows.net/",
"CosmosAccount__accountEndpoint": "https://<your-cosmos-account>.documents.azure.com:443/",
"ServiceBusAccount__fullyQualifiedNamespace": "<your-service-bus>.servicebus.windows.net",
"AzureAd__Instance": "https://login.microsoftonline.com/",
"AzureAd__TenantId": "<your-tenant-id>",
"AzureAd__ClientId": "<your-client-id>",
"AzureAd__Audience": "<your-api-audience>"

Running the App

1. Ensure your Azure CLI context has the necessary Role-Based Access Control (RBAC) permissions (e.g., Storage Blob Data Contributor, Cosmos DB Built-in Data Contributor, Service Bus Data Owner).
2. Run the application:

func start

Current Status & TODOs

• DocumentProcessingTrigger.cs: The polling loop for Azure Document Intelligence is currently a work in progress. It successfully submits the processing job but requires the final JSON extraction and Cosmos DB patch logic to be completed.
• Redis Caching: Implementation of Azure Cache for Redis on the GET endpoints is pending to fulfill the optimization requirements.
• APIM Policy Configuration: The Bicep/Terraform scripts for deploying the APIM rate limiting and JWT validation policies need to be finalized.