Tensors

If you have reached here, you probably know what a tensor is, and probably have heard many jokes about what a tensor is^[1]. Nevertheless, we are gonna give a brief remainder.

A tensor $T$ of order^[2] $n$ is a multilinear^[3] function between $n$ vector spaces over a field $\mathcal{F}$.

$$T:{\mathcal{F}}^{\dim (1)}\times \cdots \times {\mathcal{F}}^{\dim (n)}\mapsto \mathcal{F}$$

In layman's terms, you can view a tensor as a linear function that maps a set of vectors to a scalar.

$$T({\mathbf{v}}^{(1)},\dots ,{\mathbf{v}}^{(n)})=c\in \mathcal{F}\mathrm{\forall }i,{\mathbf{v}}^{(i)}\in {\mathcal{F}}^{\dim (i)}$$

Just like with matrices and vectors, $n$-dimensional arrays of numbers can be used to represent tensors. Furthermore, scalars, vectors and matrices can be viewed as tensors of order 0, 1 and 2, respectively.

The dimensions of the tensors are usually identified with labels and known as tensor indices or just indices. By appropeately fixing the indices in a expression, a lot of different linear algebra operations can be described.

For example, the trace operation...

$$tr(A)=\sum _i{A}_{ii}$$

... a tranposition of dimensions...

$$A_{ji}=A_{ij}^T$$

... or a matrix multiplication.

$$C_{ik}=\sum _j{A}_{ij}{B}_{jk}$$

The Tensor type

In Tenet, a tensor is represented by the Tensor type, which wraps an array and a list of index names. As it subtypes AbstractArray, many array operations are automatically dispatched.

You can create a Tensor by passing an AbstractArray and a Vector or Tuple of Symbols.

julia> Tᵢⱼₖ = Tensor(rand(3,5,2), (:i,:j,:k))
3×5×2 Tensor{Float64, 3, Array{Float64, 3}}:
[:, :, 1] =
 0.519866  0.8257    0.92294   0.928543   0.591423
 0.553337  0.982412  0.603269  0.0778786  0.943424
 0.161217  0.287726  0.591934  0.962693   0.0745474

[:, :, 2] =
 0.852252  0.82994   0.645339   0.534029  0.861693
 0.588007  0.603822  0.0692267  0.943862  0.0735332
 0.992818  0.983484  0.429221   0.909241  0.368919

Use parent and inds to access the underlying array and indices respectively.

julia> parent(Tᵢⱼₖ)
3×5×2 Array{Float64, 3}:
[:, :, 1] =
 0.519866  0.8257    0.92294   0.928543   0.591423
 0.553337  0.982412  0.603269  0.0778786  0.943424
 0.161217  0.287726  0.591934  0.962693   0.0745474

[:, :, 2] =
 0.852252  0.82994   0.645339   0.534029  0.861693
 0.588007  0.603822  0.0692267  0.943862  0.0735332
 0.992818  0.983484  0.429221   0.909241  0.368919

julia> inds(Tᵢⱼₖ)
(:i, :j, :k)

The dimensionality or size of each index can be consulted using the size function.

julia> size(Tᵢⱼₖ)
(3, 5, 2)

julia> size(Tᵢⱼₖ, :j)
5

julia> length(Tᵢⱼₖ)
30

Note that these indices are the ones that really define the dimensions of the tensor and not the order of the array dimensions.

julia> a = Tensor([1 0; 1 0], (:i, :j))
2×2 Tensor{Int64, 2, Matrix{Int64}}:
 1  0
 1  0

julia> b = Tensor([1 1; 0 0], (:j, :i))
2×2 Tensor{Int64, 2, Matrix{Int64}}:
 1  1
 0  0

julia> a ≈ b
true

This is key for interacting with other tensors.

julia> c = a + b
2×2 Tensor{Int64, 2, Matrix{Int64}}:
 2  0
 2  0

julia> parent(a) + parent(b)
2×2 Matrix{Int64}:
 2  1
 1  0

As such adjoint doesn't permute the dimensions; it just conjugates the array.

julia> d = Tensor([1im 2im; 3im 4im], (:i, :j))
2×2 Tensor{Complex{Int64}, 2, Matrix{Complex{Int64}}}:
 0+1im  0+2im
 0+3im  0+4im

julia> d'
2×2 Tensor{Complex{Int64}, 2, Matrix{Complex{Int64}}}:
 0-1im  0-2im
 0-3im  0-4im

julia> conj(d)
2×2 Tensor{Complex{Int64}, 2, Matrix{Complex{Int64}}}:
 0-1im  0-2im
 0-3im  0-4im

Contraction

Einsum operations are performed automatically with contract. Unlike other tensor libraries, the einsum pattern is not explicitly stated by the user but implicitly inferred from the Tensor objects; i.e. repeated indices will be contracted while unique indices will remain. However, the user might require some flexibility on the output and contracted indices. That's why contract has two extra keyword arguments: dims, which lists the indices to be contracted, and out, which lists the resulting indices after the contraction. Keep in mind that you're not forced to define them: dims defaults to the repeated indices and out defaults to the unique indices, but it's not recommended to define both.

For example, let's imagine that we want to perform the following operation: A sum over one dimension of a tensor.

$$X_j=\sum _i{A}_{i}j$$

contract can act on just one tensor (unary contraction) and the user can write the following operation in two different ways:

julia> contract(a; dims=[:i])
2-element Tensor{Int64, 1, Vector{Int64}}:
 2
 0

julia> contract(a; out=[:j])
2-element Tensor{Int64, 1, Vector{Int64}}:
 2
 0

For the case of binary contraction, imagine the following matrix multiplication:

$$Y_j=\sum _i{A}_{i}j{B}_{j}i$$

Then the default would be enough, although you can still define dims or out.

julia> contract(a, b)
0-dimensional Tensor{Int64, 0, Array{Int64, 0}}:
2

julia> contract(a, b; dims=[:i])
2-element Tensor{Int64, 1, Vector{Int64}}:
 2
 0

julia> contract(a, b; out=[:j])
2-element Tensor{Int64, 1, Vector{Int64}}:
 2
 0

But what if instead of contracting index :i, we want to perform a Hadamard product (element-wise multiplication)? Then that's a case where implicit inference of the einsum rule is not enough and you need to specify dims or out.

julia> contract(a, b; dims=Symbol[])
2×2 Tensor{Int64, 2, Matrix{Int64}}:
 1  0
 1  0

julia> contract(a, b; out=[:i,:j])
2×2 Tensor{Int64, 2, Matrix{Int64}}:
 1  0
 1  0

Indexing

Tensor, as a subtype of AbstractArray, allows direct indexing of the underneath array with getindex/setindex! or the [...] notation.

julia> a[1,1] = 3
3

julia> a[1,:]
2-element Vector{Int64}:
 3
 0

But like explained above, on Tensor you should refer the dimensions by their index label, which Tenet allows in many methods.

julia> a[i=1,j=1]
3

Check out that not specifying all the indices is equivalent to using : on the non-specified indices.

julia> a[i=1]
2-element Vector{Int64}:
 3
 0

julia> a[i=1,j=:]
2-element Vector{Int64}:
 3
 0

Other supported methods are permutedims, selectdim and view.

julia> permutedims(a, [:j, :i])
2×2 Tensor{Int64, 2, Matrix{Int64}}:
 3  1
 0  0

julia> selectdim(a, :i, 1)
2-element Tensor{Int64, 1, SubArray{Int64, 1, Matrix{Int64}, Tuple{Int64, Base.Slice{Base.OneTo{Int64}}}, true}}:
 3
 0

julia> view(a, :i=>1)
2-element Tensor{Int64, 1, SubArray{Int64, 1, Matrix{Int64}, Tuple{Int64, Base.Slice{Base.OneTo{Int64}}}, true}}:
 3
 0

DocumenterMermaid.MermaidScriptBlock([...])

---

1. For example, recursive definitions like a tensor is whatever that transforms as a tensor. ↩︎

2. The order of a tensor may also be known as rank or dimensionality in other fields. However, these can be missleading, since it has nothing to do with the rank of linear algebra nor with the dimensionality of a vector space. Thus we prefer to use the word order. ↩︎

3. Meaning that the relationships between the output and the inputs, and the inputs between them, are linear. ↩︎