Camera Calibration

Camera Calibration

Wanho Choi

(wanochoi.com)

• How to solve Ax=b : http://wanochoi.com/lecture/Ax=b.pdf • Least Squares : http://wanochoi.com/lecture/Least_Squares.pdf • Transformation Matrix : http://wanochoi.com/lecture/TransformationMatrix.pdf • Rotation : http://wanochoi.com/lecture/Rotation.pdf

Preliminaries

How to capture? [Idea #1]

film

(photon sensor)

the captured image

no image fully blurred

object

the sun

How to capture? [Idea #2]

the captured image object film barrier a small hole (known as aperture) (photon sensor) upside-down sharp

but, insufficient light

requires large exposure time

the sun

How to capture? [Idea #3]

film

lens (photon sensor)

the captured image

upside-down sharp sufficient light object the sun (light source)

Convex Lens Formula

H h a b f α α β β tanα = H a = hb tanβ = H_f = h_{b − f} h H = ba h H = b − ff = bf − 1 ∴ b a = bf − 1 ⇒ ⇒ b f = ba + 1 = a + ba ⇒ ∴ 1_f = a + b_ab = 1_a + 1_b upside-down upright : focal length focal point tanα = H a tanα = h_b tanβ = h_{b − f} tanβ = H f optical center

Aperture (

조리개)

• 렌즈(lens)로 들어오는 광량(amount of light)을 조절한다.

• 또한, 피사계 심도(depth-of-field)도 이에 영향을 받는다.

: 초점이 맞은 것으로 인식되는 (acceptably sharp) 거리의 범위

• 구멍의 크기가 작을수록 핀홀 카메라(pinhole camera)에 가까워 진다.

Aperture (

조리개)

Aperture (

조리개)

Ideal Thin Lens vs Real Lens

spherical aberration chromatic aberration

https://expertphotography.com/chromatic-aberration-photography/ https://en.wikipedia.org/wiki/Spherical_aberration http://www.drewgrayphoto.com/learn/distortion101

• Pinhole as a point

• Pencil of rays: 모든 광선(ray)은 한 점(focal point)을 통과

• One ray per each point

Pinhole Camera

image plane pinhole virtual image

• “어두운 방”이라는 뜻

• 그림 등을 그리기 위해 만든 광학 장치로, 사진술의 전신

Camera Obscura

Camera Obscura

Homogeneous Coordinate System

• 유클리디언 기하학(Euclidean geometry)에서 사용하는 좌표계 : 데카르트 좌표계 (cartesian coordinate system)

• _{사영 기하학(projective geometry)에서 사용하는 좌표계}

: 동차 좌표계 (homogeneous coordinate system)

(x, y, z) (x, y, z,1) (x, y, z, w)

→

→

→

4x4 or 3x4 matrix_computation

→

(x/w, y/w, z/w)

Homogeneous Coordinate

• The 2D point (x, y) is represented by the homogeneous coordinate (x, y,1).

• In general, the homogeneous coordinate (x, y, w) represents the 2D point (x/w, y/w).

Pinhole Camera Model

X_C Y_C Z_C 1 = r₁₁ r₁₂ r₁₃ t₁ r₂₁ r₂₁ r₂₃ t₂ r₃₁ r₃₂ r₃₃ t₃ 0 0 0 1 X Y Z 1 ˜x ˜y ˜z = f 0 0 0 0 f 0 0 0 0 1 0 X_C Y_C Z_C 1 (x, y) = (˜x/˜z, ˜y/˜z) ˜u ˜v ˜w = s_x s_θ u_c 0 s_y v_c 0 0 1 ˜x ˜y ˜z (u, v) = (˜u/˜w , ˜v/˜w ) camera space (3D) ➔ image plane space (2D)

world space (3D) ➔ camera space (3D)

image plane space (2D) ➔ pixel space (sensor) (2D)

y z O = C0 : optical center : optical axis P(XC, YC, ZC) p = ? f: focal length image plane p = [xy] ≡ ˜x ˜y ˜z = f XC/ZC f YC/ZC 1 ≡ f Xf YCC ZC = 0 f 0 0f 0 0 0 0 0 1 0 XC YC ZC 1 pixel scale image center skewness

(usually negligible or zero)

rotation translation projection ˜x ˜y ˜z = f 0 0 0 0 f 0 0 0 0 1 0 X_C YC Z_C 1 f 0 0 0 f 0 0 0 1 [ 1 0 0 0 0 1 0 0 0 0 1 0] zooming 3D to 2D

: standard (or canonical) projection matrix

Pinhole Camera Model

˜u

˜v

˜

w

=

s

_x

s

_θ

u

_c

0 s

_y

v

_c

0 0 1

f 0 0 0

0 f 0 0

0 0 1 0

r

₁₁

r

₁₂

r

₁₃

t

₁

r

₂₁

r

₂₁

r

₂₃

t

₂

r

₃₁

r

₃₂

r

₃₃

t

₃

0 0 0 1

X

Y

Z

1

Pinhole Camera Model

˜u

˜v

˜

w

=

s

_x

s

_θ

u

_c

0 s

_y

v

_c

0 0 1

f 0 0 0

0 f 0 0

0 0 1 0

r

₁₁

r

₁₂

r

₁₃

t

₁

r

₂₁

r

₂₁

r

₂₃

t

₂

r

₃₁

r

₃₂

r

₃₃

t

₃

0 0 0 1

X

Y

Z

1

K: intrinsic parameters W: extrinsic parameters

Pinhole Camera Model

˜u

˜v

˜

w

=

s

_x

s

_θ

u

_c

0 s

_y

v

_c

0 0 1

f 0 0 0

0 f 0 0

0 0 1 0

r

₁₁

r

₁₂

r

₁₃

t

₁

r

₂₁

r

₂₁

r

₂₃

t

₂

r

₃₁

r

₃₂

r

₃₃

t

₃

0 0 0 1

X

Y

Z

1

K: intrinsic parameters W: extrinsic parameters

C: camera matrix

K =

α γ u

_c

0 β v

_c

0 0 1

=

s

_x

s

_θ

u

_c

0 s

_y

v

_c

0 0 1

f 0 0 0

0 f 0 0

0 0 1 0

Homography

• Transformation between two different planes

• Homography matrix

‣ 3x3 square matrix

‣ But, 8 DoF as it is estimated up to a scale

‣ It is generally with . s x′y′ 1 = H [ x y 1] = h₁₁ h₁₂ h₁₃ h₂₁ h₂₂ h₂₃ h₃₁ h₃₂ h₃₃ [ x y 1] h₃₃ = 1

The Examples of Homography

DLT (Direct Linear Transformation)

˜u

˜v

˜

w

=

α γ u

_c

0 β v

_c

0 0 1

r

₁₁

r

₁₂

r

₁₃

t

₁

r

₂₁

r

₂₁

r

₂₃

t

₂

r

₃₁

r

₃₂

r

₃₃

t

₃

0 0 0 1

X

Y

Z

1

11 unknowns (11 D.O.F.) rx, ry, rz, tx, ty, tz α, β, γ, u5 unknownsc, vc 6 unknowns observed image point (measure) known control point (given)

DLT (Direct Linear Transformation)

C: camera matrix

˜u

˜v

˜

w

=

α γ u

_c

0 β v

_c

0 0 1

r

₁₁

r

₁₂

r

₁₃

t

₁

r

₂₁

r

₂₁

r

₂₃

t

₂

r

₃₁

r

₃₂

r

₃₃

t

₃

0 0 0 1

X

Y

Z

1

3 × 1

p = CP

3 × 4 4 × 1

DLT (Direct Linear Transformation)

p = CP

˜u

˜v

˜

w

=

C

₁₁

C

₁₂

C

₁₃

C

₁₄

C

₂₁

C

₂₂

C

₂₃

C

₂₄

C

₃₁

C

₃₂

C

₃₃

C

₃₄

X

Y

Z

1

u = ˜u_˜w = _CC11X + C12Y + C13Z + C14 31X + C32Y + C233Z + C34 v = ˜v_˜w = _CC21X + C22Y + C23Z + C24 31X + C32Y + C33Z + C34 So, we need at least 6 point pairs.

DLT (Direct Linear Transformation)

u = C_C11X + C12Y + C13Z + C14 31X + C32Y + C33Z + C34 v = C_C21X + C22Y + C23Z + C24 31X + C32Y + C33Z + C34 [−X −Y −Z −1 00 0 0 0 −X −Y −Z −1 vX vY vZ v]0 0 0 uX uY uZ u C₁₁ C₁₂ C₁₃ C₁₄ C₂₁ C₂₂ C₂₃ C₂₄ C₃₁ C₃₂ C₃₃ C₃₄ = 0

DLT (Direct Linear Transformation)

• For N-point pairs

−X₁ −Y₁ −Z₁ −1 0 0 0 0 u₁X₁ u₁Y₁ u₁Z₁ u₁ 0 0 0 0 −X₁ −Y₁ −Z₁ −1 v₁X₁ v₁Y₁ v₁Z₁ v₁ −X₂ −Y₂ −Z₂ −1 0 0 0 0 u₂X₂ u₂Y₂ u₂Z₂ u₂ 0 0 0 0 −X₂ −Y₂ −Z₂ −1 v₂X₂ v₂Y₂ v₂Z₂ v₂ ⋮ ⋮ −X_N −Y_N −Z_N −1 0 0 0 0 u_NX_N u_NY_N u_NZ_N u_N 0 0 0 0 −X_N −Y_N −Z_N −1 v_NX_N v_NY_N v_NZ_N v_N C₁₁ C₁₂ C₁₃ C₁₄ C₂₁ C₂₂ C₂₃ C₂₄ C₃₁ C₃₂ C₃₃ C₃₄ = 0 12N × 12 12 × 1

Mc = 0

DLT (Direct Linear Transformation)

Mc = 0 Mc = w

!

̂c = argmin

c

(w

T

_w)

wTw = (Mc)T(Mc) = cTMTMc = cT_(USVT₎T _(USVT_{) c}

= cT_(VSUT_{) (USV}T_{) c = c}T_VSUT_USVT_c

= cTVS2VTc = cT ( 12 ∑ i=1 s_i2v_iv_iT ) c

: SVD (Singular Value Decomposition)

: the 12th (=smallest) eigenvector of V

= cT _(s2

1v1v1T + s22v2v2T + ⋯ + s122 v12v12T ) c

∴ ̂c = v_{12 ( ∵ v}T_i v_{j = 0)}

If individual parameters are needed

C = [H|h] = KR [I| − C

0

] = H [I| − C

0

]

known

H = QR

: QR decomposition

∴ K = 1

Q

₃₃

Q

h = − HC

₀

∴ C

₀

= − Hh

: homogeneity normalization

• Checkerboard

‣ Size & structure are known.

‣ Easy to set & get the points.

Camera Calibration using 2D Pattern

• All points are on a plane, so Z=0.

x

y

z

Camera Calibration using 2D Pattern

• All points are on a plane, so Z=0.

• We cannot solve the problem with general DLT process.

−X₁ −Y₁ −Z₁ −1 0 0 0 0 u₁X₁ u₁Y₁ u₁Z₁ u₁ 0 0 0 0 −X₁ −Y₁ −Z₁ −1 v₁X₁ v₁Y₁ v₁Z₁ v₁ −X₂ −Y₂ −Z₂ −1 0 0 0 0 u₂X₂ u₂Y₂ u₂Z₂ u₂ 0 0 0 0 −X₂ −Y₂ −Z₂ −1 v₂X₂ v₂Y₂ v₂Z₂ v₂ ⋮ ⋮ −X_N −Y_N −Z_N −1 0 0 0 0 u_NX_N u_NY_N u_NZ_N u_N 0 0 0 0 −X_N −Y_N −Z_N −1 v_NX_N v_NY_N v_NZ_N v_N C₁₁ C₁₂ C₁₃ C₁₄ C₂₁ C₂₂ C₂₃ C₂₄ C₃₁ C₃₂ C₃₃ C₃₄ = 0 rank deficiency!

A Simple Trick!

˜u

˜v

˜

w

=

α γ u

_c

0 β v

_c

0 0 1

r

₁₁

r

₁₂

r

₁₃

t

₁

r

₂₁

r

₂₁

r

₂₃

t

₂

r

₃₁

r

₃₂

r

₃₃

t

₃

0 0 0 1

X

Y

Z

1

˜u

˜v

˜

w

=

α γ u

_c

0 β v

_c

0 0 1

r

₁₁

r

₁₂

t

₁

r

₂₁

r

₂₁

t

₂

r

₃₁

r

₃₂

t

₃

[

X

Y

1]

H = [h

₁

, h

₂

, h

₃

] = K[r

₁

, r

₂

, t]

8 unknowns (8 D.O.F.)

Homography

• Linear transformation between two different planes

p = HP

[

u

v

1]

= [h

1

, h

2

, h

3

] [

X

Y

1]

[

u

v

1]

=

h

₁₁

h

₁₂

h

₁₃

h

₂₁

h

₂₂

h

₂₃

h

₃₁

h

₃₂

h

₃₃

[

X

Y

1]

observed image point (measure) known control point (given) 8 unknowns

= 1

So, we need at least 4 point pairs.

How to get K, R, and T from H

H = [h

₁

, h

₂

, h

₃

] = K[r

₁

, r

₂

, t]

rotation matrix가 아니기 때문에 QR decomposition 사용 불가능

r

₁

= K

−1

h

₁

r

₂

= K

−1

h

₂

r

T₁

r

₂

= 0

r

T 1

r

1

= r

T2

r

2

= 1

(K−1h1)T(K−1h2) = 0 이 관계로 부터 다음과 같이 2개의 제약조건(constraints)을 얻을 수 있다. (K−1h1)T(K−1h1) = (K−1h2)T(K−1h2) hT₁K−TK−1h₂ = 0 hT₁K−TK−1h₁ − hT₂K−TK−1h₂ = 0

∴

How to get K, R, and T

hT₁K−TK−1h₂ = 0

hT

1K−TK−1h1 − hT2K−TK−1h2 = 0

: symmetric positive definite matrix

B = b₁₁ b₁₂ b₁₃ b₂₁ b₂₂ b₂₃ b₃₁ b₃₂ b₃₃ ∵ B = K−T_K−1 _{= (K}−T_{) (K}−T₎T _{= AA}T _{: Cholesky decomposition} ∴ b := [b11 b12 b13 b22 b23 b33]T B := K−T_K−1 h T 1Bh2 = 0 hT 1Bh1 − hT2Bh2 = 0

How to get K, R, and T

hT₁K−TK−1h₂ = 0 hT 1K−TK−1h1 − hT2K−TK−1h2 = 0 B := K−T_K−1 h T 1Bh2 = 0 hT 1Bh1 − hT2Bh2 = 0 b := [b11 b12 b13 b22 b23 b33]T vT 12b = 0 (v11 − v22)T b = 0 vT_{ij = [h}1ih1j h1ih2j + h2ih1j h3ih1j + h1ih3j h2ih2j h3ih2j + h2ih3j h3ih3j]T [ vT 12 (v11 − v12)T] b = 0

How to get K, R, and T

the 1st point pair

the n-th point pair

vT₁₂ (v11 − v12)T ⋯ vT 12 (v11 − v12)T b = 0

Vb = 0

Vb = 0 Vb = w

!

̂b = argmin

b

(w

T

_w)

→

→

If individual parameters are needed

H = [h

₁

, h

₂

, h

₃

] = K[r

₁

, r

₂

, t]

r

1

= K

−1

h

1

r

₂

= K

−1

h

₂

B = K

−T

_K

−1 : Cholesky decomposition

B = K

−T

K

−1

= AA

T

∴ K = A

−T

r

₁

= K

−1

_h

1

&

r

2

= K

−1

h

2

h

₃

= Kt → t = K

−1

h

₃

• Distortion: non-linear error

• Especially radial distortion

• So, we only consider the first two terms of radial distortion.

• The distortion function is dominated by the radial components, and especially dominated by the first term.

• Moreover, more elaborated model would cause numerical instability.

Dealing with Radial Distortion

˘x = x + x [k1(x2 + y2)2+k2(x2 + y2)2]

˘y = y + y [k1(x2 + y2)2+k2(x2 + y2)2] (˘x, ˘y)

(x, y): ideal (distortion-free) point : real (distorted) point

˜u ˜v 1 = α 0 u_c 0 β v_c 0 0 1 ˜x ˜y 1 ˜u = u_c + α ˘x ˜v = v_c + α˘y ˘u = u + (u − u0) x [k1(x2 + y2)2+k2(x2 + y2)2] ˘v = v + (v − v0) x [k1(x2 + y2)2+k2(x2 + y2)2] [(u − u 0) (x2 + y2) (u − u0) (x2 + y2) (v − v0) (x2 + y2) (v − v0) (x2 + y2)] [ k₁ k₂] = [˘v − v]˘u − u

Maximum Likelihood Estimation

• Non-linear optimization problem

• Levenberg-Marquardt algorithm

• Initial guess from DLT

N

∑

i=1 M

∑

j=1

p

_ij

− ˘p

_ij

_{(K, k}

₁

, k

₂

, R

_i

, t

_i

, P

_j

₎

2

Camera Calibration using OpenCV

References

• https://www.youtube.com/watch?reload=9&v=vZELygPzV0M • https://www.youtube.com/watch?v=ywternCEqSU • https://www.youtube.com/watch?v=Ou9Uj75DJX0 • http://staff.fh-hagenberg.at/burger/publications/reports/ 2016Calibration/Burger-CameraCalibration-20160516.pdf • https://www.microsoft.com/en-us/research/wp-content/uploads/ 2016/02/tr98-71.pdf