Involving Auto Regression (AR) into ANN

We apply ANN by involving the non-linear activation function $\phi$ into the standard Auto-Regression (AR) model

$$w_t = \phi (\sum_{i=1}^p a_i w_{t-i})+\epsilon_t.$$ (30)

The idea lurking behind ANN-AR model is that the activation function $\phi$ roughly represents the activation of a real neuron. We minimize the sum

$$f_m(x)=\sum_{t=1}^T \epsilon_t^2,$$ (31)

where the objective $f_m(x)$ depends on $l$ unknown parameters represented as a $l$-dimensional vector $x=(x_k, k= 1,...,p)=(a_i, i=1,...,p)$.

From expression (1.31) it is clear that the residuals $\epsilon_t$ are nonlinear functions of parameters $a_t$ if the activation function $\phi$ is nonlinear . This means that the minimum conditions

$${\partial f_m(x) \over \partial a_i} =0,\ i=1,...,p$$ (32)

is a system of nonlinear equations that may have a multiple solution.

An interesting activation function is derived using the Gaussian distribution function

$$\phi(w_t(l))=\frac{\beta}{\sqrt{2\phi}\sigma}\int_{-\infty}^{w_t(l)} e^{-\frac{w-\mu}{\sigma}} dw$$ (33)

Here $w_t(l)=\sum_{i=1}^l a_i w_{t-i}$ and $\beta$ is a scale parameter. The function (1.34) is different from the activation of a real neuron but is convenient for analysis.

Using activation function (1.34) sum (1.31) depends on the parameters $\beta,\ \sigma, \ \mu$, too. These parameters are unknown, as usual. Therefore they should be optimized together with the parameters $a$. In such a case the objective $f_m(x)$ depends on $p+3$ unknown parameters represented as a $p+3$-dimensional vector $x=(x_k, k= 1,...,p+3)=(a_i, i=1,...,p, \beta, \sigma, \mu)$.

That is the main difference of model (1.34) from the traditional ANN models where the activation functions are selected by their resemblance to the natural ones defined using the biophysical experimentation. In this research the resemblance factor is neglected and the activation function (1.34) is regarded just as a reasonable non-linearity that should be adapted to the available data.

Obviously, using ANN one meets the multi-modality problem as it was in a case of ARMA model (see non-linear equations (1.11)). The multi-modality problems of ANN models are discussed in [].