In this work, we introduce compositional tensor networks (CTN), as a new approximation format merging the strengths of low-rank tensor formats and neural networks. Tensor networks have gained prominence in high-dimensional data analysis and functional approximation, particularly for their robustness and computational efficiency. Neural networks, while more powerful, often require extensive resources and time for training.
CTNs combine the benefits of both approaches. We also propose a training procedure for this architecture based on the natural gradient descent. The natural gradient is known for being invariant under reparameterization and for aligning with the true functional gradient under an appropriate metric. This frequently leads to faster and more stable convergence compared to standard Euclidean-gradient-based optimization. Although computing the natural gradient is generally intractable in high-dimensional settings, we show that in the CTN context, it can be computed efficiently using tensor algebra. Moreover, the structure of CTNs allows the use of low-rank constraints during training, further improving computational scalability. The efficiency of this approach is illustrated on benchmark problems in regression.