**Camera Calibration**

**Wanho Choi**

**(wanochoi.com)**

• **How to solve Ax=b**
: http://wanochoi.com/lecture/Ax=b.pdf
• **Least Squares**
: http://wanochoi.com/lecture/Least_Squares.pdf
• **Transformation Matrix**
: http://wanochoi.com/lecture/TransformationMatrix.pdf
• **Rotation**
: http://wanochoi.com/lecture/Rotation.pdf

**Preliminaries**

**How to capture? [Idea #1]**

**film**

**(photon sensor)**

**the captured image**

**no image**
**fully blurred**

**object**

**the sun**

**How to capture? [Idea #2]**

**the captured image**
**object**
**film**
**barrier**
**a small hole**
**(known as aperture)**
**(photon sensor)**
**upside-down**
**sharp**

**but, insufficient light**

**requires large exposure time**

**the sun**

**How to capture? [Idea #3]**

**film**

**lens** **(photon sensor)**

**the captured image**

**upside-down**
**sharp**
**sufficient light**
**object**
**the sun**
**(light source)**

**Convex Lens Formula**

*H*

*h*

*a*

*b*

*f*

*α*

*α*

*β*

*β*

*tanα = H*

*a*

*= hb*

*tanβ = H*

_{f}*= h*

_{b − f}*h*

*H*

*= ba*

*h*

*H*

*= b − ff*

*= bf*− 1

*∴ b*

*a*

*= bf*− 1 ⇒ ⇒

*b*

*f*

*= ba*

*+ 1 = a + ba*⇒ ∴ 1

_{f}*= a + b*= 1

_{ab}*+ 1*

_{a}

_{b}**upside-down**

**upright**: focal length focal point

*tanα = H*

*a*

*tanα = h*

_{b}*tanβ = h*

_{b − f}*tanβ = H*

*f*optical center

**Aperture (**

**조리개)**

• **렌즈(lens)로 들어오는 광량(amount of light)을 조절한다.**

• **또한, 피사계 심도(depth-of-field)도 이에 영향을 받는다.**

: 초점이 맞은 것으로 인식되는 (acceptably sharp) 거리의 범위

• **구멍의 크기가 작을수록 핀홀 카메라(pinhole camera)에 가까워 진다.**

**Aperture (**

**조리개)**

**Aperture (**

**조리개)**

**Ideal Thin Lens vs Real Lens**

**spherical aberration** **chromatic aberration**

https://expertphotography.com/chromatic-aberration-photography/ https://en.wikipedia.org/wiki/Spherical_aberration http://www.drewgrayphoto.com/learn/distortion101

• Pinhole as a point

• Pencil of rays: 모든 광선(ray)은 한 점(focal point)을 통과

• One ray per each point

**Pinhole Camera**

image
plane
pinhole
virtual
image
• “어두운 방”이라는 뜻

• 그림 등을 그리기 위해 만든 광학 장치로, 사진술의 전신

**Camera Obscura**

**Camera Obscura**

**Homogeneous Coordinate System**

• **유클리디언 기하학(Euclidean geometry)에서 사용하는 좌표계**
: 데카르트 좌표계 (cartesian coordinate system)

• _{사영 기하학(projective geometry)에서 사용하는 좌표계}

: 동차 좌표계 (homogeneous coordinate system)

*(x, y, z) (x, y, z,1) (x, y, z, w)*

### →

### →

### →

4x4 or 3x4 matrix_{computation}

### →

*(x/w, y/w, z/w)*

**Homogeneous Coordinate**

• The 2D point (x, y) is represented by the homogeneous coordinate (x, y,1).

• In general, the homogeneous coordinate (x, y, w) represents the 2D point (x/w, y/w).

**Pinhole Camera Model**

*X*

_{C}*Y*

_{C}*Z*1 =

_{C}*r*

_{11}

*r*

_{12}

*r*

_{13}

*t*

_{1}

*r*

_{21}

*r*

_{21}

*r*

_{23}

*t*

_{2}

*r*

_{31}

*r*

_{32}

*r*

_{33}

*t*

_{3}0 0 0 1

*X*

*Y*

*Z*1

*˜x*

*˜y*

*˜z*=

*f 0 0 0*

*0 f 0 0*0 0 1 0

*X*

_{C}*Y*

_{C}*Z*1

_{C}*(x, y) = (˜x/˜z, ˜y/˜z)*

*˜u*

*˜v*˜

*w*=

*s*

_{x}*s*

_{θ}*u*

_{c}*0 s*

_{y}*v*0 0 1

_{c}*˜x*

*˜y*

*˜z*

*(u, v) = (˜u/˜w , ˜v/˜w )*

**camera space (3D) ➔ image plane space (2D)**

**world space (3D) ➔ camera space (3D)**

**image plane space (2D) ➔ pixel space (sensor) (2D)**

*y*
*z*
*O = C*0
: optical center
: optical axis
*P(XC, YC, ZC*)
*p = ?*
*f*: focal length
image plane
*p = [xy] ≡*
*˜x*
*˜y*
*˜z* =
*f XC/ZC*
*f YC/ZC*
1
≡ *f Xf YCC*
*ZC*
= *0 f 0 0f 0 0 0*
0 0 1 0
*XC*
*YC*
*ZC*
1
**pixel**
**scale**
**image**
**center**
**skewness**

**(usually negligible or zero)**

**rotation** **translation**
**projection**
*˜x*
*˜y*
*˜z* =
*f 0 0 0*
*0 f 0 0*
0 0 1 0
*X _{C}*

*YC*

*Z*1

_{C}*f 0 0*

*0 f 0*0 0 1 [ 1 0 0 0 0 1 0 0 0 0 1 0]

**zooming**

**3D to 2D**

**: standard (or canonical)**
** projection matrix**

**Pinhole Camera Model**

*˜u*

*˜v*

### ˜

*w*

### =

*s*

_{x}*s*

_{θ}*u*

_{c}*0 s*

_{y}*v*

_{c}### 0 0 1

*f 0 0 0*

*0 f 0 0*

### 0 0 1 0

*r*

_{11}

*r*

_{12}

*r*

_{13}

*t*

_{1}

*r*

_{21}

*r*

_{21}

*r*

_{23}

*t*

_{2}

*r*

_{31}

*r*

_{32}

*r*

_{33}

*t*

_{3}

### 0 0 0 1

*X*

*Y*

*Z*

### 1

**Pinhole Camera Model**

*˜u*

*˜v*

### ˜

*w*

### =

*s*

_{x}*s*

_{θ}*u*

_{c}*0 s*

_{y}*v*

_{c}### 0 0 1

*f 0 0 0*

*0 f 0 0*

### 0 0 1 0

*r*

_{11}

*r*

_{12}

*r*

_{13}

*t*

_{1}

*r*

_{21}

*r*

_{21}

*r*

_{23}

*t*

_{2}

*r*

_{31}

*r*

_{32}

*r*

_{33}

*t*

_{3}

### 0 0 0 1

*X*

*Y*

*Z*

### 1

**K: intrinsic parameters** **W: extrinsic parameters**

**Pinhole Camera Model**

*˜u*

*˜v*

### ˜

*w*

### =

*s*

_{x}*s*

_{θ}*u*

_{c}*0 s*

_{y}*v*

_{c}### 0 0 1

*f 0 0 0*

*0 f 0 0*

### 0 0 1 0

*r*

_{11}

*r*

_{12}

*r*

_{13}

*t*

_{1}

*r*

_{21}

*r*

_{21}

*r*

_{23}

*t*

_{2}

*r*

_{31}

*r*

_{32}

*r*

_{33}

*t*

_{3}

### 0 0 0 1

*X*

*Y*

*Z*

### 1

**K: intrinsic parameters** **W: extrinsic parameters**

**C: camera matrix**

**K =**

*α γ u*

_{c}*0 β v*

_{c}### 0 0 1

### =

*s*

_{x}*s*

_{θ}*u*

_{c}*0 s*

_{y}*v*

_{c}### 0 0 1

*f 0 0 0*

*0 f 0 0*

### 0 0 1 0

**Homography**

• Transformation between two different planes

• Homography matrix

‣ 3x3 square matrix

‣ **But, 8 DoF as it is estimated up to a scale**

‣ It is generally with .
*s* *x′y′*
1 **= H** [
*x*
*y*
1] =
*h*_{11} *h*_{12} *h*_{13}
*h*_{21} *h*_{22} *h*_{23}
*h*_{31} *h*_{32} *h*_{33} [
*x*
*y*
1]
*h*_{33} = 1

**The Examples of Homography**

**DLT (Direct Linear Transformation)**

*˜u*

*˜v*

### ˜

*w*

### =

*α γ u*

_{c}*0 β v*

_{c}### 0 0 1

*r*

_{11}

*r*

_{12}

*r*

_{13}

*t*

_{1}

*r*

_{21}

*r*

_{21}

*r*

_{23}

*t*

_{2}

*r*

_{31}

*r*

_{32}

*r*

_{33}

*t*

_{3}

### 0 0 0 1

*X*

*Y*

*Z*

### 1

**11 unknowns (11 D.O.F.)**

*rx, ry, rz, tx, ty, tz*

*α, β, γ, u*

**5 unknowns**

*c, vc*

**6 unknowns**observed image point (measure) known control point (given)

**DLT (Direct Linear Transformation)**

**C: camera matrix**

*˜u*

*˜v*

### ˜

*w*

### =

*α γ u*

_{c}*0 β v*

_{c}### 0 0 1

*r*

_{11}

*r*

_{12}

*r*

_{13}

*t*

_{1}

*r*

_{21}

*r*

_{21}

*r*

_{23}

*t*

_{2}

*r*

_{31}

*r*

_{32}

*r*

_{33}

*t*

_{3}

### 0 0 0 1

*X*

*Y*

*Z*

### 1

3 × 1**p = CP**

3 × 4 4 × 1
**DLT (Direct Linear Transformation)**

**p = CP**

*˜u*

*˜v*

### ˜

*w*

### =

*C*

_{11}

*C*

_{12}

*C*

_{13}

*C*

_{14}

*C*

_{21}

*C*

_{22}

*C*

_{23}

*C*

_{24}

*C*

_{31}

*C*

_{32}

*C*

_{33}

*C*

_{34}

*X*

*Y*

*Z*

### 1

*u = ˜u*

_{˜}

*=*

_{w}*11*

_{C}C*X + C*12

*Y + C*13

*Z + C*14 31

*X + C*32

*Y + C*233

*Z + C*34

*v = ˜v*

_{˜}

*=*

_{w}*21*

_{C}C*X + C*22

*Y + C*23

*Z + C*24 31

*X + C*32

*Y + C*33

*Z + C*34

**So, we need at least 6 point pairs.**

**DLT (Direct Linear Transformation)**

*u =*

*C*11

_{C}*X + C*12

*Y + C*13

*Z + C*14 31

*X + C*32

*Y + C*33

*Z + C*34

*v =*

*C*21

_{C}*X + C*22

*Y + C*23

*Z + C*24 31

*X + C*32

*Y + C*33

*Z + C*34 [

*−X −Y −Z −1 0*0 0 0

*0 −X −Y −Z −1 vX vY vZ v]*0 0

*0 uX uY uZ u*

*C*

_{11}

*C*

_{12}

*C*

_{13}

*C*

_{14}

*C*

_{21}

*C*

_{22}

*C*

_{23}

*C*

_{24}

*C*

_{31}

*C*

_{32}

*C*

_{33}

*C*

_{34}= 0

**DLT (Direct Linear Transformation)**

• *For N-point pairs*

*−X*_{1} *−Y*_{1} *−Z*_{1} −1 0 0 0 *0 u*_{1}*X*_{1} *u*_{1}*Y*_{1} *u*_{1}*Z*_{1} *u*_{1}
0 0 0 *0 −X*_{1} *−Y*_{1} *−Z*_{1} *−1 v*_{1}*X*_{1} *v*_{1}*Y*_{1} *v*_{1}*Z*_{1} *v*_{1}
*−X*_{2} *−Y*_{2} *−Z*_{2} −1 0 0 0 *0 u*_{2}*X*_{2} *u*_{2}*Y*_{2} *u*_{2}*Z*_{2} *u*_{2}
0 0 0 *0 −X*_{2} *−Y*_{2} *−Z*_{2} *−1 v*_{2}*X*_{2} *v*_{2}*Y*_{2} *v*_{2}*Z*_{2} *v*_{2}
⋮
⋮
*−X _{N}*

*−Y*

_{N}*−Z*−1 0 0 0

_{N}*0 u*

_{N}X_{N}*u*

_{N}Y_{N}*u*

_{N}Z_{N}*u*0 0 0

_{N}*0 −X*

_{N}*−Y*

_{N}*−Z*

_{N}*−1 v*

_{N}X_{N}*v*

_{N}Y_{N}*v*

_{N}Z_{N}*v*

_{N}*C*

_{11}

*C*

_{12}

*C*

_{13}

*C*

_{14}

*C*

_{21}

*C*

_{22}

*C*

_{23}

*C*

_{24}

*C*

_{31}

*C*

_{32}

*C*

_{33}

*C*

_{34}= 0

*12N × 12 12 × 1*

**Mc = 0**

**DLT (Direct Linear Transformation)**

**Mc = 0 Mc = w**

### !

**̂c = argmin**

**̂c = argmin**

**c**

**(w**

*T*

_{w)}

_{w)}

**w**

*T*

**w = (Mc)**

*T*

**(Mc) = c**

*T*

**M**

*T*

**Mc**

**= c**

*T*

_{(USV}*T*

_{)}

*T*

_{(USV}*T*

_{) c}**= c***T*_{(VSU}*T*_{) (USV}*T*_{) c = c}*T*_{VSU}*T*_{USV}*T*_{c}

**= c***T***VS**2**V***T***c**
**= c***T*
(
12
∑
*i=1*
*s _{i}*2

*v*)

_{i}v_{i}T**c**

: SVD (Singular Value Decomposition)

**: the 12th (=smallest) eigenvector of V**

**= c***T* * _{(s}*2

1*v*1*v*1*T* *+ s*22*v*2*v*2*T* *+ ⋯ + s*122 *v*12*v*12*T* **) c**

**∴ ̂c = v**_{12} _{( ∵ v}*T _{i}*

**v**

_{j}_{= 0)}

**If individual parameters are needed**

**C = [H|h] = KR [I| − C**

0**] = H [I| − C**

0### ]

known**H = QR**

: QR decomposition **∴ K = 1**

*Q*

_{33}

**Q**

**h = − HC**

_{0}

**∴ C**

_{0}

**= − Hh**

: homogeneity normalization
• **Checkerboard**

‣ Size & structure are known.

‣ Easy to set & get the points.

**Camera Calibration using 2D Pattern**

• All points are on a plane, so Z=0.

*x*

*y*

*z*

**Camera Calibration using 2D Pattern**

• All points are on a plane, so Z=0.

• We cannot solve the problem with general DLT process.

*−X*_{1} *−Y*_{1} *−Z*_{1} −1 0 0 0 *0 u*_{1}*X*_{1} *u*_{1}*Y*_{1} *u*_{1}*Z*_{1} *u*_{1}
0 0 0 *0 −X*_{1} *−Y*_{1} *−Z*_{1} *−1 v*_{1}*X*_{1} *v*_{1}*Y*_{1} *v*_{1}*Z*_{1} *v*_{1}
*−X*_{2} *−Y*_{2} *−Z*_{2} −1 0 0 0 *0 u*_{2}*X*_{2} *u*_{2}*Y*_{2} *u*_{2}*Z*_{2} *u*_{2}
0 0 0 *0 −X*_{2} *−Y*_{2} *−Z*_{2} *−1 v*_{2}*X*_{2} *v*_{2}*Y*_{2} *v*_{2}*Z*_{2} *v*_{2}
⋮
⋮
*−X _{N}*

*−Y*

_{N}*−Z*−1 0 0 0

_{N}*0 u*

_{N}X_{N}*u*

_{N}Y_{N}*u*

_{N}Z_{N}*u*0 0 0

_{N}*0 −X*

_{N}*−Y*

_{N}*−Z*

_{N}*−1 v*

_{N}X_{N}*v*

_{N}Y_{N}*v*

_{N}Z_{N}*v*

_{N}*C*

_{11}

*C*

_{12}

*C*

_{13}

*C*

_{14}

*C*

_{21}

*C*

_{22}

*C*

_{23}

*C*

_{24}

*C*

_{31}

*C*

_{32}

*C*

_{33}

*C*

_{34}= 0

**rank deficiency!**

**A Simple Trick!**

*˜u*

*˜v*

### ˜

*w*

### =

*α γ u*

_{c}*0 β v*

_{c}### 0 0 1

*r*

_{11}

*r*

_{12}

*r*

_{13}

*t*

_{1}

*r*

_{21}

*r*

_{21}

*r*

_{23}

*t*

_{2}

*r*

_{31}

*r*

_{32}

*r*

_{33}

*t*

_{3}

### 0 0 0 1

*X*

*Y*

*Z*

### 1

*˜u*

*˜v*

### ˜

*w*

### =

*α γ u*

_{c}*0 β v*

_{c}### 0 0 1

*r*

_{11}

*r*

_{12}

*t*

_{1}

*r*

_{21}

*r*

_{21}

*t*

_{2}

*r*

_{31}

*r*

_{32}

*t*

_{3}

### [

*X*

*Y*

### 1]

**H = [h**

_{1}

**, h**

_{2}

**, h**

_{3}

**] = K[r**

_{1}**, r**

_{2}**, t]**

**8 unknowns (8 D.O.F.)**

**Homography**

• Linear transformation between two different planes

**p = HP**

### [

*u*

*v*

### 1]

**= [h**

1**, h**

2**, h**

3### ] [

*X*

*Y*

### 1]

### [

*u*

*v*

### 1]

### =

*h*

_{11}

*h*

_{12}

*h*

_{13}

*h*

_{21}

*h*

_{22}

*h*

_{23}

*h*

_{31}

*h*

_{32}

*h*

_{33}

### [

*X*

*Y*

### 1]

observed image point (measure) known control point (given)**8 unknowns**

### = 1

**So, we need at least 4 point pairs.**

**How to get K, R, and T from H**

**H = [h**

_{1}

**, h**

_{2}

**, h**

_{3}

**] = K[r**

_{1}**, r**

_{2}**, t]**

rotation matrix가 아니기 때문에
QR decomposition 사용 불가능
**r**

_{1}

**= K**

−1**h**

_{1}

**r**

_{2}

**= K**

−1**h**

_{2}

**r**

*T*

_{1}

**r**

_{2}

### = 0

**r**

*T*1

**r**

1 **= r**

*T*2

**r**

2 ### = 1

**(K**−1

**h**1)

*T*

**(K**−1

**h**2) = 0 이 관계로 부터 다음과 같이 2개의 제약조건(constraints)을 얻을 수 있다.

**(K**−1

**h**1)

*T*

**(K**−1

**h**1

**) = (K**−1

**h**2)

*T*

**(K**−1

**h**2)

**h**

*T*

_{1}

**K**

*−T*

**K**−1

**h**

_{2}= 0

**h**

*T*

_{1}

**K**

*−T*

**K**−1

**h**

_{1}

**− h**

*T*

_{2}

**K**

*−T*

**K**−1

**h**

_{2}= 0

### ∴

**How to get K, R, and T**

**h***T*_{1}**K***−T***K**−1**h**_{2} = 0

**h***T*

1**K***−T***K**−1**h**1 **− h***T*2**K***−T***K**−1**h**2 = 0

: symmetric positive definite matrix

**B =**
*b*_{11} *b*_{12} *b*_{13}
*b*_{21} *b*_{22} *b*_{23}
*b*_{31} *b*_{32} *b*_{33}
**∵ B = K***−T*** _{K}**−1

_{= (K}*−T*

_{) (K}*−T*

_{)}

*T*

_{= AA}*T*

_{: Cholesky decomposition}

*11*

**∴ b := [b***b*12

*b*13

*b*22

*b*23

*b*33]

*T*

**B := K**

*−T*

**−1**

_{K}**h**

*T*1

**Bh**2 = 0

**h**

*T*1

**Bh**1

**− h**

*T*2

**Bh**2 = 0

**How to get K, R, and T**

**h**

*T*

_{1}

**K**

*−T*

**K**−1

**h**

_{2}= 0

**h**

*T*1

**K**

*−T*

**K**−1

**h**1

**− h**

*T*2

**K**

*−T*

**K**−1

**h**2 = 0

**B := K**

*−T*

**−1**

_{K}**h**

*T*1

**Bh**2 = 0

**h**

*T*1

**Bh**1

**− h**

*T*2

**Bh**2 = 0

*11*

**b := [b***b*12

*b*13

*b*22

*b*23

*b*33]

*T*

**v**

*T*12

**b = 0**

**(v**11

**− v**22)

*T*

**b = 0**

**v**

*T*

_{ij}

_{= [h}1ih1j*h1ih2j*

*+ h2ih1j*

*h3ih1j*

*+ h1ih3j*

*h2ih2j*

*h3ih2j*

*+ h2ih3j*

*h3ih3j*]

*T*[

**v**

*T*12

**(v**11

**− v**12)

*T*]

**b = 0**

**How to get K, R, and T**

the 1st point pair

the n-th point pair

**v***T*_{12}
**(v**11 **− v**12)*T*
⋯
**v***T*
12
**(v**11 **− v**12)*T*
**b = 0**

**Vb = 0**

**Vb = 0 Vb = w**

### !

**̂b = argmin**

**̂b = argmin**

**b**

**(w**

*T*

_{w)}

_{w)}

### →

### →

**If individual parameters are needed**

**H = [h**

_{1}

**, h**

_{2}

**, h**

_{3}

**] = K[r**

_{1}**, r**

_{2}**, t]**

**r**

1 **= K**

−1**h**

1
**r**

_{2}

**= K**

−1**h**

_{2}

**B = K**

*−T*

_{K}

−1
: Cholesky decomposition
_{K}

**B = K**

*−T*

**K**

−1 **= AA**

*T*

**∴ K = A**

*−T*

**r**

_{1}

**= K**

−1_{h}

1 _{h}

### &

**r**

2 **= K**

−1**h**

2
**h**

_{3}

**= Kt → t = K**

−1**h**

_{3}

• **Distortion: non-linear error**

• **Especially radial distortion**

• **So, we only consider the first two terms of radial distortion.**

• The distortion function is dominated by the radial components, and especially dominated by the first term.

• Moreover, more elaborated model would cause numerical instability.

**Dealing with Radial Distortion**

*˘x = x + x [k*1*(x*2 *+ y*2)2*+k*2*(x*2 *+ y*2)2]

*˘y = y + y [k*1*(x*2 *+ y*2)2*+k*2*(x*2 *+ y*2)2] (*˘x, ˘y)*

*(x, y)*: ideal (distortion-free) point
: real (distorted) point

*˜u*
*˜v*
1 =
*α 0 u _{c}*

*0 β v*0 0 1

_{c}*˜x*

*˜y*1

*˜u = u*

_{c}*+ α ˘x*

*˜v = v*

_{c}*+ α˘y*

*˘u = u + (u − u*0

*) x [k*1

*(x*2

*+ y*2)2

*+k*2

*(x*2

*+ y*2)2]

*˘v = v + (v − v*0

*) x [k*1

*(x*2

*+ y*2)2

*+k*2

*(x*2

*+ y*2)2] [

*(u − u*0

*) (x*2

*+ y*2

*) (u − u*0

*) (x*2

*+ y*2)

*(v − v*0

*) (x*2

*+ y*2

*) (v − v*0

*) (x*2

*+ y*2)] [

*k*

_{1}

*k*

_{2}] = [

*˘v − v]˘u − u*

**Maximum Likelihood Estimation**

• Non-linear optimization problem

• Levenberg-Marquardt algorithm

• Initial guess from DLT

*N*

### ∑

*i=1*

*M*

### ∑

*j=1*

**p**

_{ij}**− ˘p**

_{ij}_{(K, k}

_{(K, k}_{1}

*, k*

_{2}

**, R**

_{i}*, t*

_{i}**, P**

_{j}