Backup for AI workloads: New rules for a new era

By Emilio Griman

Companies are investing heavily in infrastructure transformation to support AI workloads, but many are not backing up AI workloads with the same urgency.

The ramifications of neglecting to establish AI-ready backup strategies are escalating weekly as rapidly expanding cyber threats target susceptible AI workloads. A cyberattack can lead to significant financial losses; the deletion of essential models, training datasets and experimental metadata; and prolonged downtime of vital business operations such as automation, analytics and critical services. An incident can also lead to potential harm to organizational reputation and customer trust, among other issues. 

Backup procedures must also evolve as AI models grow more complex, data pipelines become more adaptable and inference shifts toward the edge.

Why is AI workload backup lagging?

A substantial attack surface, escalating complexity and additional factors make AI workloads vulnerable to cybersecurity threats. There are several reasons why the requirements for backing up AI-driven workloads render traditional backup methods inadequate.

1. Complexity
   AI workloads require large, dynamic datasets, including intermediate data, model checkpoints and lineage metadata. Backups also must account for versioned model artifacts that necessitate consistent updates to preserve data integrity. 
2. Lack of integration
   Numerous backup platforms remain inadequately optimized to support AI pipelines, containerized configurations and GPU-accelerated clusters. Distributed infrastructures frequently encompass edge devices, cloud services and hybrid systems within AI deployments.
3. Security and compliance pressures
   AI data often encounters stringent compliance regulations, and the sensitivity of data typically necessitates the use of encryption, immutability or air-gapped storage. These considerations demand more advanced backup procedures.

What are common cyber threats for AI workloads?

AI workloads’ dependence on large data sets and complex architecture exposes them to unique cybersecurity risks. The most prominent cyber threats for AI workloads are:

• Data poisoning: The injection of corrupt or malicious data into training datasets, which compromises the integrity of AI models
• Model inversion: The reconstruction of sensitive data by analyzing or investigating the outputs of an AI model
• Prompt injection: The introduction of malicious input into an AI model to alter the behavior of the model and influence outputs
• Shadow AI: The unapproved use of external AI tools within an organization, typically in conflict with official governance policies
• Supply chain attacks: The compromise of AI systems via vulnerabilities in third-party dependencies

How does Kyndryl approach AI workload backup?

Kyndryl stands out by delivering a cyber-resilient, end-to-end recovery framework designed for modern AI workloads. Kyndryl recommends the following best practices to protect AI workloads:

• Data backup and protection: Regularly create backups of training datasets, feature stores and raw data in secure storage systems to guarantee reproducibility and meet compliance requirements.
• Model checkpointing and versioning: Save intermediate model states during training and keep version control for deployed models to facilitate rollback and disaster recovery.
• Infrastructure and environment snapshots: Take snapshots of compute environments (VMs, containers) and configurations to enable quick recovery of the AI pipeline in case of failure.
• Workflow and metadata preservation: Archive orchestration workflows, hyperparameters and experiment metadata to ensure traceability and auditability.
• Automation and monitoring: Automate backup schedules according to workload criticality to ensure that mission-critical models and datasets are backed up with greater frequency and redundancy; integrate machine learning techniques to identify anomalies in backup success rates and unforeseen data drift; and use real-time telemetry dashboards for immediate visibility into retention policies, redundancy levels and recovery performance.
• Recovery optimization: Regularly practice restore procedures within sandbox environments using pre-validated recovery plans to remain prepared and assured that the process will function effectively when required, and make sure recovery plans include systems like AD/LDAP that support AI orchestration.
• Governance, compliance and documentation: Maintain a reliable record of individuals who initiated backups, implemented modifications and performed restorations; ensure that backup audit trails comply with standards such as ISO 27001 and provisions of the AI Act; and convey your organization's risk posture and preparedness to executives and regulatory authorities.

This approach ensures that AI workloads remain secure, compliant and recoverable — far beyond what traditional backup solutions can offer.

Here’s what makes Kyndryl’s approach to resilient backup strategies different from traditional approaches:

• Cyber recovery focus: Moves beyond traditional backups to ensure rapid restoration and operational continuity, protecting against ransomware and data breaches
• Strategic partnerships: Integrates Kyndryl’s expertise in orchestration, automation and artificial intelligence with data protection technologies and immutable storage solutions provided by our partners
• Unified data management: Provides seamless backup, recovery and compliance across hybrid and multi-cloud environments, ensuring scalability for large AI datasets
• Zero trust and immutability: Implements advanced security principles to safeguard critical AI data against evolving threats
• Proactive cyber-resiliency: Integrates incident recovery services and orchestration tools to minimize downtime and maintain business continuity