numerical gradient with extra-large data size

Hi:

I have a 3D coordinate with significantly large size: 1e9*3.

and I have value of parameter at each of these points such as T: 1e9*1.

now I need the gradient of T at each direction, such as dT/dx, dT/dy, dT/dz.

is there anyway to do this?

Thanks!

Li

Answer 1

[Walter Roberson]

You might be able to take advantage of "tall arrays" https://www.mathworks.com/help/matlab/tall-arrays.html

[Yu Li]

Hi: the memory in my PC is 96GB, so it should be able to accormodate such a large size. the major problem here is how to calculate the gradient of a parameter. could you please give me some more suggestions?

Thanks! Li

[Walter Roberson]

If you have enough memory for the temporary arrays, then you can just calculate the same way as you would if the data were smaller, by calling gradient() with three outputs. https://www.mathworks.com/help/matlab/ref/gradient.html

[Yu Li]

Hi:

Thanks for your reply.

The problem here is: the parameter T is not a function of the coordinate, and the space is not uniform between each points. So looks the ‘gradient’ function does not work here.

Thanks!

LI

[Image Analyst]

If you turn the array into a 3-D image you could use convn().

[Yu Li]

Hi:

Thanks for your reply.

Could you please give me some more suggestions? I have no idea how the convolution could solve this problem.

Thanks!

Li

[Walter Roberson]

Is it correct that you have a set of scattered points that are not at regular intervals in the coordinates, and you want to calculate the gradient? If so then do you want to calculate the gradient over a grid or only at the existing points?

With scattered points it will be necessary to use an interpolation method. Is (bi-)linear interpolation acceptable or do you need something like spline ?

[Yu Li]

Hi:

Yes. You are right. I have set of points with 3D coordinates and corresponding parameter such as temperature.

I need the gradient over The Who points.

The linear interpolation is acceptable.

I have tried the interpolation method. 1. Generate a 3D coordinates which cover these points and interpolate them.

However, since the number of base coordinates are too large (over 50 million), it requires significantly large amount time and memory to do it.

Do you have any other suggestions?

Thanks!

Li

[Walter Roberson]

See https://projecteuclid.org/download/pdf_1/euclid.rmjm/1250127676 for a discussion of algorithms, and http://www.tandfonline.com/doi/pdf/10.1080/02626667409493918 for more information on the one they recommend.

But I wonder what you are headed for?

http://journals.ametsoc.org/doi/abs/10.1175/1520-0493%281994%29122%3C1611%3AUOMIFM%3E2.0.CO%3B2 "Use of Multiquadric Interpolation for Meteorological Objective Analysis "

http://www.worldscientific.com/worldscibooks/10.1142/6437 "Meshfree Approximations in MATLAB"

[Yu Li]

Hi:

thanks for your reply. I will read these papers and see if I could find any solution.

the original data sets are come from a high fedelity CFD code using large eddy simulation (LES), the CFD software only exports the velocity but no velocity gradient. but we need the gradient to do further analysis.

Thanks!

Li