If I understood your request correctly...
1.) You know the four points of the original rectangle:
A, B, C, D (where I assume the Z component is always 0)
2.) You know the four points of the rotated rectangle (in 3D space):
A', B', C', D'
3.) I assume, you know the rotation matrices for 3D Spaces
E.g.
Rotation matrix - Wikipedia[
^]
4.) You are looking for the three rotation angles α, β, γ.
Now we can write down all the equations (based on the rotation matrices) that needs to be fulfilled for the above and you will recognize there will be much more equations, then the variables you are looking for.
Conclusion: That means, there will be no unique solution.
In other words: an infinite number of combinations of angles α, β, γ will give a solution
Btw:
I'm always fascinated by linear algebra but I'm far from being a professional here.
@Stefan63 (
Stefan_Lang - Professional Profile[
^]) can explain this much better and also provide the mathematical conclusions/proofs ;)
Submit an article or tip