IMAGE EXTRAPOLATION USING SPARSE METHODS IMAGE EXTRAPOLATION USING SPARSE METHODS

representation. Second one is Morphological Component Analysis (MCA) which is based on Independent Component Analysis (ICA). The last is the Expectation Maximization (EM) algorithm. This algorithm is statistics-based. These three algorithms for image extrapolation are compared on the real images.


Introduction
Image processing using underdetermined systems of linear equations is a very promising approach in different applications.In the previous years methods for image denoising based on image splitting were used [1].Today, we could denoise the image via underdetermined systems of linear equations [2].Except image denoising we use these systems for image compression [3], deblurring [4] or image inpainting [5] and for many other applications.
We will use the algorithms originally designed for image inpainting to image extrapolation.Image inpainting is the framework for filling in the known holes in the input image.Classical inpainting methods assume filling the holes from different directions.If we want to use these algorithms for image extrapolation we must redefine or modify some parameters or parts of the original algorithms.
If we are using underdetermined systems of linear equations we want to find the sparse solution.The sparse solution is the one which contains only a few nonzero coefficients.The basics of sparse signal representations are introduced in [3].All methods for finding sparse solutions start from the basic problem (P 0 ) which is defined as follows: subject to y ϭ Dx (1) where y is the known signal (e.g.image) we want to reconstruct, D is the dictionary which consists of n atoms ("elementary signals") in columns and m rows which denote the length of atoms.An unknown sparse solution represents amounts of each atom in the original signal.We define the norm of a vector: and 0 Ͻ p Ͻ ϱ.If 0 Ͻ p Ͻ 1, it is actually not the norm, it is the quasinorm.The quasinorm is similar to the standard norm, but it does not satisfy the triangle inequality.
The optimization problem (1) is defined in ᐍ 0 .When ʈ x ʈ 0 ϽϽ n for x ʦ ‫ޒ‬ n , we could say that x is sparse.Searching for the sparse solution without any algorithm in ᐍ 0 norm is NP-hard problem.The problem can be redefined to: where ε is the error of solution.There are various algorithms dealing with this problem.The overview of these algorithms is presented in [6].

A) K-SVD algorithm
In most applications via underdetermined systems, we assume fixed dictionary D. K-SVD algorithm is the algorithm for adaptive learning the dictionary that allows sparse representation of the input signal.It is called K-SVD because of the SVD (Singular Value Decomposition) algorithm which is always performed K times where K is the count of columns in D. K-SVD algorithm is the generalization of the K-means algorithm [7].K-SVD is the iterative algorithm containing two basic steps.The first one is finding the

Jan Spirik -Jan Zatyik *
Image extrapolation is the specific application in image processing.You have to extrapolate the image for example when you want to process the given image piecewise.When the border patches are incompleted you must extrapolate them to the given size.Nowadays,some basic extrapolations, e.g.linear, polynomial etc. are used.The advanced methods are presented in this paper.We are using the algorithms that are based on finding the sparse solutions in underdetermined systems of linear equations.Three algorithms are presented for image extrapolation.First one is the K-SVD algorithm.K-SVD is the algorithm that trains a dictionary which allows the optimal sparse representation.Second one is Morphological Component Analysis (MCA) which is based on Independent Component Analysis (ICA).The last is the Expectation Maximization (EM) algorithm.This algorithm is statistics-based.These three algorithms for image extrapolation are compared on the real images.
sparse solution x for the actual dictionary.The second step is the dictionary update.Before application of these steps we must initialize the dictionary.The initialized dictionary must have all atoms (all columns) ᐍ 2 normalized.We could use, for example, DCT dictionary or otherwise a random matrix for the initial dictionary.
We could formulate the K-SVD as: where Y is a matrix that contains training samples {y i } N iϭ1 in columns, X is a matrix that contains the corresponding coefficients, T 0 is the error of the representation and F denotes the Frobenius norm.The Frobenius norm is defined as: , In the first stage we assume D fixed.We can express the penalty term as: , Then we could divide (6) into problems: These problems could be solved by the known algorithms for finding sparse solutions in underdetermined systems of linear equations.
In the second stage the dictionary D is updated.We assume D and X fixed.In each step only one atom (column in D) d k and corresponding row x k T in X will be updated.Based on previous statement we define sets ω k that consists of indexes of vectors {y i } which use the atom d k , in fact where x k T is nonzero: Then we define matrices of errors E k that express error for all N samples with missing k-atom: With these conditions we could rewrite (4) as: (10) For simplification we apply sets ω k to the matrices E k in that manner we choose only columns that correspond with ω k and we get E R k .Using the algorithm SVD we divide E R k .into:

Y DX y Dx
The last step in the dictionary update stage is only replacing the current atom (column) of the dictionary d k with the first column of the matrix U and the corresponding row of coefficients x k T with the first column of matrix V multiplied by .
Extrapolation via K-SVD is realized as training the dictionary on the image patches with the chosen size (for example 8 ϫ 8, 16 ϫ 16 pixels).The count of atoms in the dictionary is chosen by user too.For the image extrapolation we add random pixels in the place where we want to extrapolate the image.We also define the binary matrix M which extends the original image.Ones in the binary mask represent the known pixels in the image and zeros indicate the missing ones.We can reformulate problem (P 0 ) as: subject to MDx ϭ y. (12) If we perform K-SVD we apply M to the dictionary D, but in the extrapolated part of the image we use the whole trained dictionary.For better results we completely overlap all the patches from the image [3].
One of the biggest disadvantages of K-SVD algorithm is the dependency on the length of extrapolation with the patch size.We must train the dictionary for the patches of minimum (e ϩ 1) ϫ ϫ (e ϩ 1) pixels, where e is the length of extrapolation in pixels.It means that the computational efficiency is dependent quadratically on the length of extrapolation.This patch size is chosen because of the overlapping patches.If we do not fulfill this limit, the extrapolation will be unsuccessful, the end of the extrapolated part will be black.
On the other hand, the big advantage of K-SVD extrapolation is that the training part for one image is done only once.You can extrapolate the image to different directions, but the trained dictionary remains the same.Only the reconstruction will be done.For example, if you train the dictionary for patches 8 ϫ 8 pixels, you should perform extrapolation to any directions with the extrapolation length of maximum 7 pixels.

B) Morphological Component Analysis
The basics of Morphological Component Analysis (MCA) are introduced in [8].We assume that the input picture is a linear combination of two independent parts: cartoon and texture.This idea comes from Independent Component Analysis (ICA).We use two incoherent dictionaries.The dictionary D t allows sparse decomposition of the part of the image y t (we assume the picture as 1-D vector) that contains only texture.On the opposite side, the dictionary D c allows sparse decomposition of the part of the image y c that contains only cartoon.We could formulate the problem as: subject to Because of the problem for finding the numerical solution we must reformulate equation (13) as: where the third term of the equation reflects the reconstruction error and TV is the abbreviation for total variation penalty.We use the TV for recovering piecewise smooth objects with pronounced edges, when applied to the cartoon layer [9].The total variation is essentially the ᐍ 1 norm of the gradient.
For the purpose of image extrapolation, we must apply the binary mask M the same way as for K-SVD to the image: Ones in the binary mask express the known pixels in the image and zeros indicate the missing ones.
If we assume that y t ϭ D t x t and we know the texture part of the image y t , we can calcute the sparse vector x t as x t ϭ D t ϩ x t ϩ r t , where r t is a residual vector in the null-space of D t and ϩ denotes Moore-Penrose pseudoinverse.We apply the same properties to the cartoon part.For simplification we assume r t ϭ r c ϭ 0. Then we could minimize the problem as: (16) When we implement the algorithm we first choose the threshold factor, number of iterations and the parameters λ and γ.The last two parameters should be constant during the iterations or they could be descended.Then we initialize the cartoon part of the image as y c ϭ y and the texture part as y t ϭ 0. After that we iterate the algorithm to the stopping rule (threshold or number of iterations).In the iteration part of the algorithm we first fix the texture part of the image y t and we update the cartoon part y c and then vice-versa.After these two stages we apply the TV penalization.
The important step in the algorithm is the choice of the dictionaries.For texture part we should use local DCT, Gabor or wavelet packets transforms and for the cartoon part wavelet, curvelet, ridgelet, contourlet and many other transforms [9].MCA extrapolation significantly depends on the choice of dictionaries.For different types of pictures (real-life pictures, cartoons, computer images) a different combination of dictionaries is more vital.The extrapolation is performed for every single direction separately.You cannot do any temporary calculations.You only perform the reconstructions phase, there is no training phase as for K-SVD.All the conditions for successful extrapolation are included in the algorithm.In this case we use this method for finding sparse solutions based on penalized maximum-likelihood estimator estimation [5].In terms of statistics inpainting is a problem of estimation from incomplete data sets.EM algorithm is used for computation of sparse vector x from the previous iteration.We reconstruct the whole image, not only the missing parts.As well as MCA the efficiency of EM depends on the choice of the dictionary.One of the biggest advantages is that we could make a dictionary from several different transformations, i.e. utilize underdetermination of the system.

C) Expectation-Maximization algorithm
At first we must define the penalized maximum-likelihood estimator [5]: where σ 2 is a variance of zero-mean additive white Gaussian noise, λ Ͼ 0 is regularization parameter and Ψ is a penalty function.The penalty function must be non-negative, continuous, even-symmetric and non-decreasing function.But it must not be necessarily convex in ‫ޒ‬ ϩ .We often use ᐍ 1 -norm penalty for Ψ.We divide the input image into two sets: y 0 which contains the known pixels from the image and y m which contains the unknown pixels from the image.The incomplete observation makes impossible calculate (17) at once.We use EM algorithm for iterative reconstruction of missing data so that we use (17) for computation of new estimations until the convergence is achieved.
As already mentioned above the EM algorithm is iterative process.It consists of two steps: Expectation (E) step and Maximization (M) step.In the first step the conditional expectation of the penalized log-likelihood of complete data y 0 and actual parameters is calculated.The particular expectations can be expressed as the conditional expected squared values of the missing data: We could express the estimation at t iteration as: (20) where y act ϭ My and M is the binary mask with the same properties as above.
In M step we maximize the penalized function with the missing observations using the estimates from the E step at t iteration: where n 0 is the number of observed pixels.
These two steps are repeated t times until the convergence is reached: that is the maximum-likelihood estimate of the noise variance inside the mask with the observed pixels.The properties of convergence depend on the structure of the dictionary.The algorithm behaves differently when the dictionary is the basis, tight frame or union of several incoherent orthogonal dictionaries [5].
The EM extrapolation is in properties very similar to MCA extrapolation.There is no training phase.The result depends on the dictionary if it is basis or union of basis or etc.
The biggest disadvantage of EM algorithm is that the reconstruction is performed on the whole image, not only on planned parts.It means the extrapolated image is blurred.The blur effect depends on the length of extrapolation.You can improve the EM reconstruction result by some technique presented in [4].

Experimental results
We choose the excerpt of the Barbara image (Fig. 1) to compare the efficiency of the presented algorithms.The algorithms were compared for different length and type of extrapolation.For the purposes of experimental measurements, the binary mask M to the known part of the image was applied.It is because of possibility to R E V I E W measure the efficiency of the algorithms.The binary mask will be always rectangular.We choose the length e of extrapolation and then which direction we want to extrapolate.We could make extrapolation for more directions at once.We fill in e columns or rows (it depends on the directions of extrapolation) with zeros in M.
The standard PSNR and SSIM are applied for measurement reconstruction quality.We are introducing some examples of extrapolation in the following figures (Figs. 2 -5).For all extrapolations using MCA and EM algorithms two incoherent dictionaries were used: curvelets [10] and undecimated discrete wavelet transform [11].

Conclusion
The principles of three different algorithms for image extrapolation that are using sparse solutions have been presented.The results for different lengths and directions of extrapolation were shown in the figures.Our subjective perception of the quality of the reconstructed images corresponded with the objective measurement of quality: PSNR and SSIM.The best algorithm from the presented algorithms was K-SVD algorithm.The quality was especially in good reconstruction of the texture parts in the image.There is also only a small blur effect.Some modifications can be made in the original K-SVD, for example, non-square patches and so on.These and more modifications will be presented in the future work.

Acknowledgement
The described research was performed in laboratories supported by the SIX project; the registration number CZ.
Expectation-Maximization algorithm (EM) is based on mathematical statistics.It is iterative method for estimation of maximum-