Tatiana - the problem is that a is initialized as a scalar but it is treated as an array at the line
A= [A, x, a(x), b(x), z];
This is true for b as well. So if they should be arrays, then where is the code to update them? Also, the lines
cell (size(a)) = (a+(abs((sin(x.*2-1.5))^2+(3*sin(x)))+2.38));
cell (size(b))= b+((cos(x))^2*sin(x.*2-1)+4.29);
will cause a variable named **cell* to be created. But this is also the name of the built-in MATLAB function cell. I strongly recommend that you rename this variable to something else so that it doesn't cause other bugs with your code.