Beyond Jenkins Logs: Benchmarking MCP-Enabled AI for Root-Cause Analysis and Remediation in Enterprise CI/CD

Authors

Keywords:

Continuous integration, software reliability, human-in-the-loop, remediation, AIOps, root-cause analysis, continuous delivery, Model Context Protocol, Jenkins

Abstract

Background: Diagnosis of failed CI/CD pipelines may require evidence from deployment state, credentials, registries and infrastructure systems that is absent from Jenkins console logs.

Objective: To compare Model Context Protocol (MCP)-enabled AI with log-only AI and manual triage for root-cause diagnosis, remediation recommendations, successful pipeline reruns and diagnostic efficiency.

Methods: An incident-matched secondary analysis evaluated 320 unique Jenkins failure incidents across three assessment modes (960 evaluations) and 16 root-cause categories. Paired binary outcomes were compared using exact McNemar tests and incident-level bootstrap confidence intervals; diagnosis times were compared within incidents.

Results: MCP-enabled AI achieved 93.75% correct root-cause diagnoses, compared with 62.50% for log-only AI and 83.75% for manual triage. Remediation correctness was 91.56%, 57.19% and 80.63%, respectively, while successful reruns occurred in 70.94%, 39.06% and 59.38%. Mean diagnosis times were 8.09, 18.92 and 39.47 minutes. Fifteen MCP cases were blocked by a policy gate.

Conclusion: MCP-enabled evidence retrieval was associated with better recorded outcomes in this benchmark. Unverified incident provenance, model settings, tool traces and adjudication procedures limit generalisability and preclude claims of established effectiveness or safety in live deployments.

Downloads

Published

2026-11-21

Similar Articles

1-10 of 28

You may also start an advanced similarity search for this article.