Beyond Jenkins Logs: Benchmarking MCP-Enabled AI for Root-Cause Analysis and Remediation in Enterprise CI/CD
Keywords:
Continuous integration, software reliability, human-in-the-loop, remediation, AIOps, root-cause analysis, continuous delivery, Model Context Protocol, JenkinsAbstract
Background: Diagnosis of failed CI/CD pipelines may require evidence from deployment state, credentials, registries and infrastructure systems that is absent from Jenkins console logs.
Objective: To compare Model Context Protocol (MCP)-enabled AI with log-only AI and manual triage for root-cause diagnosis, remediation recommendations, successful pipeline reruns and diagnostic efficiency.
Methods: An incident-matched secondary analysis evaluated 320 unique Jenkins failure incidents across three assessment modes (960 evaluations) and 16 root-cause categories. Paired binary outcomes were compared using exact McNemar tests and incident-level bootstrap confidence intervals; diagnosis times were compared within incidents.
Results: MCP-enabled AI achieved 93.75% correct root-cause diagnoses, compared with 62.50% for log-only AI and 83.75% for manual triage. Remediation correctness was 91.56%, 57.19% and 80.63%, respectively, while successful reruns occurred in 70.94%, 39.06% and 59.38%. Mean diagnosis times were 8.09, 18.92 and 39.47 minutes. Fifteen MCP cases were blocked by a policy gate.
Conclusion: MCP-enabled evidence retrieval was associated with better recorded outcomes in this benchmark. Unverified incident provenance, model settings, tool traces and adjudication procedures limit generalisability and preclude claims of established effectiveness or safety in live deployments.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Harish Jangam (Author)

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Authors retain copyright. The license identified on this article governs its reuse. Under CC BY-NC-ND 4.0, users may share unadapted material for non-commercial purposes with appropriate attribution, a license link, and retention of applicable notices. Sharing adapted material and commercial reuse are not authorized by this license; seek separate permission where required. Third-party content may carry separate terms. Existing Version of Record licenses are not changed retrospectively by updates to journal policy. Full license: https://creativecommons.org/licenses/by-nc-nd/4.0/.