How to detect the orientation of an object in 3D

Dear computer vision gurus,

I have a question regarding finding the orientation of an object. I would like to know the orientation of an object from one image. The aim is to match the coordinates with my CAD model and find the orientation of the object with respect to the camera.

An example with an arbitrary object at different orientations:

I can more or less determine the orientation of the camera if it is tilted or straight, thus maybe use the "fitgeotrans" to get rid of the projection. However, I am far away from finding the orientation of the object in space.

Open for your ideas, thank you for your time.

[Guillaume]

I'm skeptical that it can be done with just one camera, usually you need stereoscopy to understand 3D scenes, but it's not my field.

When confronted with such a problem, the first thing I'd do is a scientific literature search to see how others do it. Have you done that?

[Deniz]

There are some papers related using a single image; with ellipse fitting, histogram matching and binary connectivity. But the results acquired were having relatively low accuracy than I expect.

Three nice papers:

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.818.7409&rep=rep1&type=pdf

https://www.sciencedirect.com/science/article/pii/S1755581716300086

https://www.sciencedirect.com/science/article/pii/S0097849315000680

Answer 1

[Matt J]

I would like to know the orientation of an object from one image.

If you just want the orientation and not the translation with respect to the camera, you just need two vanishing points v1 and v2 and the camera projection matrix, P=[K*R,-K*R*C]. The 3D directions d1 and d2 obtained from

d1=(K*R)\v1
d2=(K*R)\v2

then completely determine the orientation of the object. Finding the points where the red lines converge in the image would give you a v1 and where the blue lines converge would give you v2.

Answer 2

[Matt J]

If you can extract 2D landmark points x and get their corresponding 3D points X from your CAD model, you could then use the extrinsics command

https://www.mathworks.com/help/vision/ref/extrinsics.html#bui0psk

to get both rotation and translation.

If you don't have the Computer Vision Systems Toolbox, but do have the Optimization Toolbox, you could make your own single view bundle adjustment routine with lsqnonlin pretty easily,

params0 =zeros(1,6); %initial guess (example)

params= lsqnonlin(@(params)  modelfun(params, x,X,P)  ,params0);


function residual=modelfun(params, x,X,P)
%params = [theta1,theta2,theta3, c1,c2,c3]  six degree of freedom unknowns
%x = 2xN landmark image coordinates
%X = 3xN landmark coordinates in CAD model
%P = 3x4  camera projection matrix

      p=num2cell(params;)
   [theta1,theta2,theta3, c1,c2,c3]=deal(p{:});
  
   
   R=@(t) [cosd(t), sind(t); -sind(t), cosd(t)];
   
   Rx=eye(3);Rx([2,3],[2,3])=R(theta1);
   Rx=eye(3);Rx([1,3],[1,3])=R(theta2);
   Rz=eye(3);Rx([1,2],[1,2])=R(theta3);

   Xr=Rx*Ry*Rz*X+[c1;c2;c3];  %roto-translated 3D CAD model points
   
   xr=P(:,1:3)*Xr+P(:,4); %project to 2D
    xr=xr(1:2,:)./x(3,:);
   
   residual=xr-x;
   
end