Project references from KRITIS-Audit

Starting point

Under the EU System Operation Guideline SOGL (EU 2017/1485), power plant operators report their dispatch planning: per unit and day, 96 quarter-hourly values for generation, balancing reserves, redispatch and availability. At project start, around 120,000 files and 26 GB arrived every day, set to reach 2.1 million files within a year. Every file must be validated, acknowledged to its sender and reliably passed on to forecasting and grid operation.

Solution

We built an event-driven platform on Kubernetes in Azure. .NET services import the files, and a dedicated validation service checks them via Kafka against XSD and business rules at document, time series and point level. The sender receives an IEC 62325 acknowledgement that can reject individual time series or the whole message with a reason code. Valid data is stored in PostgreSQL and distributed to downstream systems via Kafka topics and file exports.

Scaling to seventeen times the volume

• Duplicates: 92% of time series are re-sent unchanged, only with a new version number. A hash of the 96 values identifies them, so they are not stored again. At target volume this reduces the write load on paper from 105,000 to around 8,400 inserts per second, and storage from 1.36 TB to around 110 GB a day.
• Persistence: The load test located the bottleneck in row-by-row writes to a single database. We switched to bulk inserts, a time series database with compression and fast hash lookups.
• Silent failures: Very large validation responses were lost in the message broker without an error, and the affected files stayed in “validation pending”. We aligned message sizes across the entire chain, and large documents now pass through in full.
• Test data: Unit-level schedules count as inside information under REMIT. Load tests therefore run on synthetic files without duplicates, which represents the worst case.
• Forecasting: A new, lean exchange format based on Protobuf delivers one package per unit and business type, with no master data on the wire.

Result

The platform validates and acknowledges every file automatically, across five environments up to production. In the load test, 80,000 files went through in just over an hour at a success rate of around 99%, without any benefit from deduplication. Bottlenecks and savings are backed by numbers, and the architecture is designed for seventeen times the volume.

Starting point

Tunnel segments are the precast concrete elements from which tunnel tubes are assembled. A precast plant produces hundreds of them every day and inspects each one for cracks or spalling before delivery. Findings were recorded across several catalogues on a handheld scanner: type of damage, location, extent. That was tedious, and different inspectors classified the same damage differently. As a result, the quality data accumulating every day was not reliable enough for analysis.

Solution

We added an AI assessment to the inspection. The inspector photographs the damage in the Angular app, and a multimodal model in Azure OpenAI, hosted in an EU region, describes the finding. In parallel, a vector search in PostgreSQL with pgvector returns similar cases that have already been confirmed. Through this retrieval-augmented generation (RAG), the model relies on the project's own data rather than general knowledge. The C# API receives a complete damage proposal in a fixed JSON schema with a confidence value, which the inspector confirms or corrects. Several catalogues become one decision about one proposal.

Analysis and guardrails

• Weekly analysis: An AI agent with tool calling queries the existing REST APIs every week and summarises patterns across formworks and concrete batches for quality management in a single sentence, for example “Formwork 14 shows edge damage in 9 of the last 40 pours”. Without consistent classification, such correlations could not be evaluated.
• Human decides: The AI only makes proposals. In every case a human makes the decision, and the product's workflow engine keeps control of the process.
• Data protection: Photos remain in the storage of the respective customer instance, and no customer data flows into model training.
• Measurability: Every proposal and every correction is logged. From this the hit rate is measured, and prompts and few-shot examples are refined against the actual corrections.

Result

Damage assessment follows one consistent proposal rather than the reading of whoever is on shift. Every correction becomes a confirmed case for the next search, so proposals improve with the data. The use case runs in production within an existing SaaS product.

Starting point

Service technicians work in the tower or the substation, often alone and under time pressure. They need to know what a fault code means on this turbine type and which version of an instruction applies today. The documentation lives on a network drive that has grown over roughly two decades: scanned wiring diagrams without a text layer, HTML manuals with tens of thousands of pages, status lists and photos, in total several hundred thousand files from more than thirty manufacturer folders. A RAG assistant with hybrid search, a reranker and page-level citations was already running on the pilot data. The open question was whether the approach would carry the real document base.

Analysis: five break points

We surveyed the document base and reviewed the ingestion code. The existing path sent every page through a large language model before anything was filtered. Extrapolated, a full run meant several million model calls, costs in the mid four-figure euro range and a runtime of days to weeks. Three of the five break points cost money, two put safety in the field at risk.

• Format: The pipeline read PDFs only. For the largest manufacturer, though, the actual documentation is an HTML manual system, and what got indexed instead were cryptically named PDF attachments.
• Text yield: Text recognition was switched off. A fifth to a third of all pages contain no readable text and still went through the model. A single wiring diagram with 258 text-free pages produced several hundred model calls and as many empty index entries.
• Duplicates and metadata: Duplicates were not detected, about half of the files for one manufacturer. Turbine model, controller type and revision were left for the model to guess, although they are almost always in the file path.
• Discarded documents: Anything that did not fit one of four categories was deleted. That hit safety data sheets, HSE warnings and status code sheets of all things.
• Revisions: Competing revisions existed side by side, and the pipeline had no notion of supersession. A withdrawn version could reach the technician as a valid answer.

Solution

The target architecture does not replace the platform, it changes the order. A deterministic stage that works without a model goes in front of the language model. It detects duplicates, reads metadata from path and file name, resolves revisions, switches on text recognition only where needed and routes by document type. Documents it cannot classify go into a review queue rather than the bin, and status codes are looked up exactly in a table. SharePoint replaces the network drive as the source, which means incremental indexing and metadata maintained by the business. We ruled out a generic Copilot solution. It loses filtering by model, controller and revision and moves confidential manufacturer documentation out of sovereign EU operation.

Result

The client has a decision paper with a prioritised plan. The first steps do not depend on the document source and can start right away: a sample run that turns the forecast into a measurement, status code tables as the cheapest safety gain, the deterministic stage and the review queue. A smaller overlap window for text fragments alone cuts model costs by up to two thirds. The assistant stays advisory. It shows the cited passage in the manual, and the person on site makes the decision.

Starting point

A quarterly business review brings together, for the quarterly customer conversation, the status of all contracts and subscriptions, the open cases and a summary for senior management. This material used to be compiled by hand from several systems. The process repeated for every customer and every quarter, tied up time in customer success, and its depth and structure depended on who compiled it.

Solution

We built a multi-agent architecture on Kubernetes. A controller agent receives the request and distributes it to domain supervisor agents for customer success and technical quality management. The agents receive their data event-driven via an event streaming platform, evaluate their part and return the building blocks. The controller assembles them into the review draft: executive summary, case overview and contract and subscription status. The team reviews and refines this draft instead of building the review from scratch.

Operations: five silent failures

The real hurdle was not the agents but the fact that failures stayed invisible. Five patterns ran through the project, and we closed each one at its root.

• Configuration: Missing configuration keys crashed the service on startup. Configuration is now read with guarded defaults, and a missing key produces a clear message.
• Releases: A failed Helm release also uninstalled the healthy pods and took the running service down with it. Deployments now pass through a deploy gate, and a failed release can no longer replace a healthy state.
• Secrets: The secret store reported a key as present while no value existed behind it. The result was silent authentication failures. Secrets are now checked before startup for the actual value rather than the mere presence of the key.
• Models: In one environment the language model was not deployed, and the service went into a crash loop. Every environment now verifies before startup that the required model is available.
• Message queue: A missing message queue let the service start as if everything were fine. The deployment was green while every request timed out. The health check now requests a real review draft and only turns green once the entire path through the queue to the response works.

Result

The review draft is generated end to end in the test and production environments. A green deploy now means a working service, because acceptance runs through the full call. The class of silent failures that previously surfaced only in day-to-day use is closed.