Crossplane Evaluation

Decision: NOT ADOPTING

Crossplane will not be used in this stack. The lack of a plan/preview mechanism is a dealbreaker for enterprise adoption and safe infrastructure management.

Why We Evaluated It

The core problem: Terraform/OpenTofu requires re-implementing the same infrastructure concepts per platform (AWS, XCP-ng, bare metal). At thousands of nodes across multiple platforms, this is a massive maintenance burden. Crossplane's XRD/Composition model promised a unified API:

XRD: "VirtualMachine" (universal API)
  ├── Composition: AWS      → EC2 instance
  ├── Composition: XCP-ng   → XO VM
  └── Composition: bare metal → MAAS / Ansible

One API, multiple backends — teams request a "VirtualMachine" and the right composition handles it.

Strengths

CNCF Graduated (Nov 2025, v2.2) — Apache 2.0 license, top-tier maturity
Continuous drift detection — automatically reverts manual changes, unlike Terraform's on-demand plan/apply
No state file management — no remote backends, locking issues, or state corruption
Kubernetes-native — works with ArgoCD, Flux, kubectl, RBAC out of the box
XRDs/Compositions — genuine multi-platform abstraction layer, solves the "re-implement per cloud" problem
Eventual consistency — resources with complex dependencies don't get stuck like Terraform's dependency graph
Enterprise adoption — Deutsche Kreditbank, Elastic, Nike, Apple, NASA, Grafana Labs, 60+ orgs
Deutsche Kreditbank replaced Terraform; deployments went from weeks to under one hour

Dealbreaker: No Plan/Preview

The single biggest issue. Terraform's terraform plan lets operators see exactly what will change before applying. Crossplane applies changes immediately upon resource creation/modification.

Discussed in the community for 2+ years with no resolution
A Kubernetes-native solution would be a Plan CRD that shows proposed changes before approval
ArgoCD sync --dry-run is a partial workaround but only shows k8s resource diffs, not what the cloud provider will actually do underneath
For regulated environments and SRE teams at scale, change preview is non-negotiable

Possible reasons it hasn't been implemented:

The continuous reconciliation architecture may make point-in-time snapshots fundamentally hard
Upbound (commercial entity) may be reserving it for their paid platform
Or simply not prioritised

Other Significant Concerns

CRD Bloat

provider-aws installs 900+ CRDs — can make API server unresponsive for up to an hour (GitHub #2649)
Exceeds Kubernetes' recommended ~500 CRD limit
Mitigated by "Provider Families" (install per-service sub-providers) but requires careful planning

Debugging Difficulty

Errors propagate through layers: Claim → XR → Composition → Managed Resource → Provider → Cloud API
Multiple sources report debugging compositions is painful
Pipeline Inspector (alpha in v2.2) is being introduced but not production-ready

Chicken-and-Egg Problem

Crossplane runs inside Kubernetes — cannot provision the cluster it runs on
Requires a "management cluster" bootstrapped by other means (Terraform, Puppet, etc.)
If the management cluster dies, no drift detection or reconciliation runs
Recovery: applying YAMLs to a new cluster works if deterministic resource names are used, otherwise risks creating duplicate cloud resources

Cluster Loss / Immutability Concerns

State lives in etcd, not a versionable state file
No independent audit trail or easy way to diff historical states
On new cluster: resources with explicit external names get adopted; auto-named resources get duplicated
Need etcd backups as insurance, and deterministic naming everywhere

Performance at Scale

~2000 composites took 6+ minutes to reconcile on k3d (GitHub #2256)
Reconciliation interval not easily configurable globally (GitHub #5934)

YAML Limitations

No native loops, conditionals, or programming constructs
Complex compositions require changes in multiple locations

XCP-ng Provider Gap

No Crossplane provider for XCP-ng exists today
A mature Terraform provider (terraform-provider-xenorchestra) exists, maintained by Vates
Could be wrapped via Upjet to auto-generate a Crossplane provider — but nobody has done it
Would be a greenfield open-source project

Real Issues Reported

API server unresponsiveness with too many CRDs (GitHub #2649)
CRD scaling issues beyond ~500 CRDs (GitHub #2895)
GCP SQL resources randomly marked for deletion — dangerous for production databases
Reconciliation rate limiting at scale (GitHub #2256)

Conclusion

Crossplane solves a real problem (multi-platform abstraction) that we need, but the lack of plan/preview makes it unsuitable for enterprise-scale production infrastructure management. The operational concerns (CRD bloat, debugging, cluster dependency) add further risk.

We need to find an alternative approach to the multi-platform abstraction problem that Crossplane solves, while retaining plan/preview capabilities.

5.0 KiB Raw Permalink Blame History