The industry is having the wrong conversation about AI Data Infrastructure

Vishal Sinha

— 11 min read
Share

Every discussion about storage for AI seems to begin with the same set of metrics:

  • Tokens per second per GPU
  • Model-loading time
  • KV cache load time
  • Dollars per terabyte

These benchmarks are real. They matter. But they are not the most important question.

The more important question is: does the storage infrastructure understand the AI workload running on top of it?

That distinction matters because production AI is not one storage workload. It is a chain of interconnected data operations spanning models, user content, retrieval, caching, agent memory, telemetry, governance, and audit.

The cost and risk do not sit inside any one of these components. They accumulate across the workflow.

This is where the difference between a data platform and a collection of point products becomes operationally significant. A point product may optimize one stage of the workflow. A platform must make the entire workflow secure, governable, observable, and repeatable.

That is the conversation we should be having.

AI is being built on storage architectures designed for something else

The storage industry has spent decades optimizing distinct systems for distinct workloads. File storage was optimized for shared access and content-heavy applications. Object storage was built for scalable capacity and durable data. Block storage was designed for transactional systems and databases.

Each architecture remains valuable. But production AI inference does not fit cleanly into any one of them.

A single inference request can touch many different data patterns:

  • Load model and configuration metadata
  • Accept a prompt and uploaded content
  • Retrieve relevant enterprise information
  • Validate permissions
  • Read embeddings and indexes
  • Access prompt or prefix caches
  • Invoke tools and applications
  • Update agent state
  • Produce multimodal output
  • Record telemetry and audit evidence
  • Preserve provenance and lineage

These operations span high-throughput sequential reads, low-latency random access, small transactional writes, streaming data, durable capacity, and memory-bandwidth-sensitive state. They can all occur in the same request path.

The default response has been to select a specialist product for each requirement and integrate them at the application layer. That appears flexible at first. At enterprise scale, it creates an integration tax that grows with every AI workload.

To understand why, it helps to organize AI data by the role it plays in the workflow.

The seven data domains of AI inference

This is not a product taxonomy. It is a workload taxonomy. It provides a better framework for evaluating the data foundation required for enterprise AI.

Domain 1 — Platform artifacts

This domain includes operating-system images, firmware, virtual-machine images, container images, cluster-node images, drivers, and platform packages.

The data is durable, versioned, and validated through checksums or signatures. It may be read infrequently during normal operation, but it becomes highly parallel and bursty when a cluster is installed, scaled, upgraded, or recovered.

The requirement is not simply capacity. The platform must distribute trusted artifacts quickly and consistently across a large infrastructure footprint.

Domain 2 — Model artifacts

This domain includes model weights, tokenizers, adapters, fine-tuning layers, quantized variants, and optimized inference engines.

Individual artifacts may range from gigabytes to terabytes. The workload is often described as reading a large file. That is an incomplete description.

The real requirement is to hydrate an inference cluster quickly, verify that the correct model version has been loaded, support parallel access, recover cleanly from partial failures, and repeat the operation without introducing inconsistent state.

The difference between storing a model and operationalizing a model is substantial.

Domain 3 — Inference content

This is the data entering and leaving the inference system. It includes prompts, uploaded documents, images, audio, video, generated text, embeddings, and other model outputs.

The I/O characteristics vary significantly:

  • Interactive inference requires low latency.
  • Batch processing requires sustained throughput.
  • Multimodal applications require streaming.
  • Large generated artifacts require scalable capacity.
  • Sensitive content requires tenant-aware protection.

These are different workload shapes within one domain, often served by the same application.

Domain 4 — Retrieval data

Retrieval data includes source documents, parsed content, chunks, embeddings, indexes, access-control information, metadata, and provenance.

In many enterprises, this will become the largest AI data domain by capacity. It is also where correctness and security intersect.

A retrieved passage is not valid merely because it is semantically relevant. It must also be current, traceable, and accessible to the user making the request.

If the retrieval system returns a relevant chunk that the user was not authorized to access, the model has not merely produced a bad answer. The system has created a data breach.

Permissions cannot be an afterthought applied after retrieval. They must travel with the data.

Domain 5 — Inference acceleration state

This domain includes key-value cache, prefix cache, model cache, and semantic response cache.

The active working set resides as close to computation as possible — first in accelerator memory and then in system memory or local flash. Shared storage may become an overflow, recovery, or persistence tier.

The data is often ephemeral, but that does not mean it is unimportant.

A cache that is not isolated correctly can leak information between users, applications, or tenants. A stale cache can cause an application to continue using information that has already been updated, revoked, or deleted.

Cache management therefore becomes a governance problem as well as a performance problem.

Domain 6 — Agent state and memory

Agentic systems introduce a new class of persistent data:

  • Working memory
  • Episodic memory
  • Semantic memory
  • Procedural knowledge
  • Execution checkpoints
  • Tool results
  • Plans and intermediate decisions

These data types have different lifetimes and access patterns, but they must operate under a common policy framework.

An enterprise cannot have one retention model for conversations, another for agent checkpoints, another for learned memory, and a fourth for tool output — particularly when all four may contain information derived from the same user interaction.

Agent memory must inherit the platform's identity, tenancy, retention, deletion, and audit policies. Today, that policy consistency is difficult to achieve.

Domain 7 — Platform operations and governance

The final domain includes routing tables, quotas, tenant assignments, policy records, secrets, telemetry, audit events, model registries, and operational metadata.

These records are generally smaller than model or content data, but they are written frequently and may require stronger consistency and longer retention.

They determine how the AI platform behaves:

  • Which model serves a request
  • Which tenant pays for it
  • Which tools an agent may invoke
  • Which data a user may access
  • Which policies must be enforced
  • Which evidence must be retained

This operational data is not separate from the AI application. It is part of the application's control plane.

Why the integration tax compounds

A specialist architecture typically assigns a separate product or service to each major requirement:

  • High-throughput model access
  • Bulk content storage
  • Vector search
  • Caching
  • Structured metadata
  • Telemetry
  • Secrets
  • Archival retention

There is nothing inherently wrong with specialization.

The problem is that each additional system introduces another identity model, another access-control framework, another retention policy, another tenancy boundary, another operational interface, and another audit stream.

The enterprise must then connect them. For every new AI application, teams must:

  • Propagate tenant identity across every component
  • Reconcile different authorization models
  • Correlate events from multiple audit streams
  • Build lineage across incompatible metadata systems
  • Coordinate retention and deletion policies
  • Reconstruct transactions across different timestamps and identifiers
  • Validate that derived artifacts inherit the policy of their source data

This work does not disappear after the first application. The second retrieval application introduces new data. The third agent introduces new tools. The fourth business unit introduces another tenancy model. A regulatory requirement introduces another retention rule.

The integration tax does not amortize. It compounds.

This is why the third enterprise AI use case can take nearly as long to operationalize as the first. The organization is not simply deploying another model. It is rebuilding the connections across an increasingly fragmented data architecture.

Five persistence tiers — and the important operations cross all of them

The seven data domains can also be mapped to five persistence tiers.

L0 — Compute-local. Active tensors and inference cache live closest to computation. Their lifetime may range from microseconds to minutes.

L1 — Node-local. Model cache, prefix cache, working memory, and temporary execution state may live on local flash or system memory. Their lifetime may range from minutes to days.

L2 — Cluster-shared. Model artifacts, vector indexes, shared caches, and workflow checkpoints require high-throughput access across multiple nodes. Their lifetime may range from days to months.

L3 — Durable system of record. User content, conversation history, enterprise knowledge, long-term agent memory, and structured metadata must be governed and retained over longer periods. Their lifetime may range from months to years.

L4 — Archive and governance. Audit records, historical outputs, compliance evidence, immutable artifacts, and legal records may need to be retained for years or indefinitely.

The difficult operations are not contained within any one tier. They cross tiers.

When a source document changes, the platform may need to invalidate derived chunks, embeddings, cached responses, agent memories, and previous conclusions.

When a user requests deletion, the platform must locate the original content and every derived representation.

When an auditor asks why an application produced a specific response, the enterprise must trace the output across model, retrieval, identity, policy, and data versions.

These are cross-tier operations. They fall between the products in a fragmented architecture.

That is the structural weakness of the point-product approach: the most important enterprise operations are often not owned by any single component.

Three properties that define a real AI data platform

A platform is not simply a portfolio of storage services. It must provide system-level properties across every data domain.

Three properties are especially important.

1. Tenant isolation across the complete workflow

Tenant isolation means that one tenant can never access another tenant's data. That guarantee must be enforced through identity, access control, network boundaries, encryption, keys, administrative separation, and data policy. It cannot depend only on application logic.

The challenge is that AI introduces new leakage paths.

A shared prefix cache may expose state derived from another tenant's prompt. A retrieval index may return a chunk from the wrong collection. Agent memory may reuse knowledge learned while acting for a different user. An operational log may contain sensitive prompt content.

A platform that isolates only the source documents but not the cache, memory, telemetry, or derived artifacts does not provide tenant isolation.

A boundary that fails in one domain is a boundary that fails everywhere.

Isolation is therefore a system property, not a component feature.

2. Audit spanning prompt, retrieval, tools, and response

"The model generated the answer" is not a defensible explanation.

An enterprise must be able to reconstruct:

  • The prompt as it reached the model
  • The system instructions applied to the request
  • The identity and tenant context
  • The data retrieved
  • The permissions used during retrieval
  • The model and adapter version
  • The tools invoked
  • The response returned
  • The citations and source versions
  • The policies enforced during execution

Regulatory, legal, security, and customer inquiries increasingly require this level of evidence.

In a fragmented architecture, the prompt may be recorded in one system, retrieval in another, generation telemetry in a third, application output in a fourth, and compliance records in a fifth.

Reconstructing a single request requires a correlation exercise across multiple event formats, clocks, identifiers, and retention windows.

That is not an audit capability. It is a forensic project.

A data platform should make the complete request history directly queryable.

3. Cross-domain lineage

Audit answers: what happened during this request?

Lineage answers: what depends on what?

An inference response may depend on a source document, a specific version of a chunk, an embedding model, an index version, a permission decision, a model version, an adapter, a prompt template, a tool result, and a sampling configuration.

Cross-domain lineage creates the graph connecting these artifacts. It must support both directions.

Forward lineage asks: this source document changed. Which chunks, embeddings, caches, responses, agent memories, and downstream artifacts are now stale?

Backward lineage asks: this answer is incorrect. Which information was retrieved, under which permission, using which model and configuration?

This graph is difficult to build when its nodes reside in independent products with different identifiers, metadata models, and versioning systems.

Most organizations end up with partial lineage: enough for a demonstration, but not enough for a production incident or regulatory inquiry.

Four moments where the platform advantage becomes visible

These ideas become tangible in four common AI workflow moments.

Model loading

The model should be loaded from the same governed platform that validates its integrity, retains approved versions, enforces immutability where required, and monitors the artifacts for malicious change.

Model supply-chain security begins at the data layer. It should not depend on an application-level process added after deployment.

Retrieval-augmented generation

Source documents, chunks, embeddings, access policies, and provenance should remain connected.

When the retriever selects a passage, its permissions should be inherited from the source. Its version should be known. Its provenance should travel into the prompt. The final response should cite the exact source version used.

The platform should understand that the document, chunk, embedding, retrieval event, and response are related parts of the same workflow.

Agent memory

Working memory, episodic memory, semantic memory, and execution checkpoints must remain within one tenant boundary.

When a retention period expires, a user exercises a deletion right, or a tenant leaves the service, the policy should propagate across all memory forms.

There should be no forgotten-store problem. An enterprise should not have to remember every database, cache, object bucket, and index in which an agent may have left a derived copy of the user's data.

Inference audit

Prompt, retrieval, tool execution, generation, response, and citation should form one queryable record.

When an enterprise needs to understand why an AI system produced a particular answer for a particular customer on a particular day, the answer should be available through a query. It should not require six weeks of engineering reconstruction.

Applying the framework with Nutanix

The Nutanix AI data platform brings the seven domains together through NUS, NDL, and NDH.

Nutanix Unified Storage (NUS)

NUS provides file, object, and block services on a unified platform. It can support platform artifacts, model artifacts, inference content, retrieval source data, shared data for acceleration tiers, long-term agent memory, and durable audit and governance archives.

The goal is not to force every workload into one data interface. The goal is to provide multiple data services on a common infrastructure, with a consistent operational and policy foundation.

Nutanix Data Lens (NDL)

NDL provides governance, access intelligence, ransomware protection, data visibility, and policy-driven controls.

Within an AI workflow, this can help protect model and source artifacts, identify sensitive data, enforce retention and immutability, analyze access patterns, and support the evidence required for audit and lineage.

Governance becomes part of the data path rather than a separate process performed after deployment.

Nutanix Data Hub (NDH)*

Nutanix Data Hub transforms fragmented enterprise data into trusted, compliant, AI-ready assets by unifying data ingestion, curation, sensitive-data protection, metadata enrichment, zero-trust governance, knowledge graphs, and secure access for AI applications and agents.

Together, NUS, NDL, and NDH create the foundation for one platform, one tenancy model, one policy plane, one governance framework, and one lineage graph.

An access-control update, retention change, deletion request, or data takedown can propagate coherently because the platform understands the relationship between the artifacts.

The reframe

The seven-domain framework changes the AI storage conversation.

The question is no longer: which storage product produces the highest benchmark?

The question becomes: which data platform understands the complete AI workflow?

That is the better question because the workflow is what compounds.

When the platform understands the workflow, every new AI application can reuse the same identity, governance, lineage, audit, data protection, and operational foundation. The second workload becomes easier than the first. The third becomes easier than the second.

When the platform does not understand the workflow, every workload introduces another set of integrations, policies, identifiers, and operational seams. The integration tax is paid again.

Three practical steps

1. Map the architecture

Overlay the current AI environment against the seven data domains. Identify which systems support each domain and where data crosses platform boundaries.

The seams will become visible quickly.

2. Calculate the integration tax

Do not look only at storage capacity and license costs. Include the number of products, the number of identity systems, the number of policy frameworks, the number of audit streams, integration engineering, operational staffing, incident reconstruction, retention and deletion workflows, and data migration complexity.

The integration cost is often larger than the storage line item.

3. Measure workflow-level outcomes

Pilot a retrieval or agentic workload across NUS, NDL, and NDH. Measure more than throughput. Measure:

  • Tenant isolation across all seven domains
  • Time required to onboard a new workload
  • Time required to trace a response to its sources
  • Ability to invalidate derived data
  • Ability to apply retention and deletion policies
  • Ability to reconstruct a complete inference transaction
  • Ability to maintain lineage across the workflow

These capabilities determine whether enterprise AI can move from isolated experimentation to repeatable production.

Stop buying storage for AI

AI does not need another disconnected storage tier. It needs a data platform that understands models, content, retrieval, caches, agents, operations, governance, and the relationships connecting them.

Performance remains important. But performance without policy creates risk. Capacity without lineage creates uncertainty. Specialized products without a common operating model create an integration burden that grows with every workload.

The enterprise AI platform will not be defined by how fast it can read a single file. It will be defined by how coherently it can manage the complete lifecycle of AI data.

That is the shift enterprises must make:

Stop buying storage for AI. Start leveraging an AI data platform.

 

 ---

*NDH is not generally available yet. Reach out for a demo request.

The opinions expressed here are my own and do not necessarily reflect the views of Nutanix.