Invited Paper

The design of active vision system for variable view imaging of micro objects

Xiaodong Tao

^*

, Deokhwa Hong, Hyungsuck Cho

Department of Mechanical Engineering, Korea Advanced Institute of Science and Technology, Korea

ABSTRACT

Insufficient vision information such as occlusion and low resolvability is one of the important issues that limit the micromanipulation and microassembly. In this paper, we proposed the active vision system that can interact with the environment by changing optical system parameters such as spatial position, orientation and focus plane. As an optomechatronic system, the proposed system integrates a pair of wedge prism, a scanning mirror, a deformable mirror and off-the-shelf optics. The compact double wedge prisms can change the view direction, however the aberration induced by wedge prisms can be corrected by deformable mirror. Combining with a scanning mirror, active optical system can observe the micro object in different view. Owing to the orthogonality of the Zernike polynomials, the proposed deformable mirror control algorithm can correct the aberration in each Zernike mode instead of controlling each actuator, which simplifies the control issue of deformable mirror. The preliminary experiment setup was built, and initial experiments were demonstrated to investigate the validity of the concept of the proposed system.

Keywords: Active vision system, micromanipulation, microassembly, deformable mirror

1. INTRODUCTION

In recent years, the development of MEMS technologies and microbiology demonstrates the need for automated microassembly and micromanipulation because the faster throughput and high yield becomes competitive requirement.

To meet these requirements, visually guided microassembly and micromanipulation has been utilized in recent years[1][2][3]. However three-dimensional microassembly and micromanipulation are still very difficult due to many reasons. One of the important reasons is insufficient vision information of three-dimensional micro objects detected from the imaging system. Occlusion and low resolvability are two main difficulties. In occlusion case, some part of important features was occluded by another object. One microassembly example is shown in Fig. 1(a) where one micro gear needs to mate with another gear. Because of the occlusion, the visual information is insufficient to guide assembly.

Resolvability refers to the ability of a visual sensor to resolve object position and orientations [4]. One example is shown in Fig. 1 (b) where the depth of the micro gear cannot be accurately resolved when it is moving along the optical axis of the camera. For micromanipualtion, Fig.1 (c) shows the cell injection. The center of the cell, holding pipette and the tip of the injection pipette should be in the same plane. With the top view of workspace, the depth of these objects can not be accurately resolved. Although depth from focus can be used, the accuracy depends on many factors, such as depth of field, focus measure function and features on the object. The problem of insufficient vision information can also happen

*taoxd@lca.kaist.ac.kr; phone: +82-42-869-3253

Invited Paper

Optomechatronic Micro/Nano Devices and Components II, edited by Yoshitada Katagiri, Proc. of SPIE Vol. 6376, 637608, (2006) · 0277-786X/06/$15 · doi: 10.1117/12.686683

Proc. of SPIE Vol. 6376 637608-1

in many other cases when three-dimensional objects need to be observed. Fig. 1 (d) shows an example of soldering inspection, where the solder joints of J lead are occluded by chip legs.

camera

detected local image

occluded region Micro gear

detected local image

moving direction shaft

camera

(a) (b)

holding pipette

injection pipette

embryo camera detected local

image

camera detected local image

occluded solder joint

(c) (d) Fig. 1. Examples of insufficient vision information: (a) occlusion in micrassembly (b) low resolvability in microassembly

(c) low resolvability in cell injection and (d) occlusion in soldering inspection

In macro word, in order to solve these problems, active vision system had been already researched, which can interact with the environment by changing camera parameters such as spatial position, orientation, focus and zoom states according to the different task. However, in micro word, the active vision systems are often more difficult to be implemented because of the complexity of the vision system and the inherent problems of optical microscopy, such as small field of view and small depth of filed. X. Tao [5] applied active zooming in microassembly, which can supply wide field of view and large depth of field at the initial state of microassembly and high resolving power at the final state. Of the particular interest is the Adaptive Scanning Optical Microscope invented by B. Potsaid [6], which can track the moving object and keep it inside of the small sub filed of view that can compose a large effective field of view at high resolution.

In this paper, we proposed the active optical system that can change the view during observation of the micro object. It can also make autofocus and correct aberration. Owing to adaptive optics, we can use off-the-shelf optics to achieve these functions. This proposed system is particularly suitable to observe the three-dimensional micro objects and

Proc. of SPIE Vol. 6376 637608-2

dynamic micro objects. It can also be used in biological observation, surface inspection, circuit board inspection, confocal microscopy and some other applications.

This paper is organized as follows: In Sec. 2, we describe the concept of the proposed system. Section 3 presents active vision system design and simulation. Section 4 shows the experimental results of active vision system.

2. CONCEPT OF THE ACTIVE VISION SYSTEM FOR VARIABLE VIEW IMAGING

2.1 Operation of the active vision system

The proposed active vision system consists of scanning mirror, wedge prisms, deformable mirror and imaging lenses.

The system layout is shown in Fig. 2.

object plane wedge prism Scanning mirror

deformable mirror

1^stImage plane

system aperture

1^stlens 2^stlens

3^stlens

4^stlens camera

θ_s

Y

X Z

θ_r

Fig. 2. Active vision system layout

The 1st Lens and 2nd Lens were used to integrate the deformable mirror to the system for wave front correction. The 3rd lens relays the active aperture of deformable mirror to the system aperture. The 4th lens will form image on the CCD camera. In the proposed system, the combination of the double wedge prisms and scanning mirror make it possible to observe the object at different view angle. The angles of scanning mirror and wedge prisms are defined as θs and θr as shown in Fig. 2. θr1 and θr1 are defined as the angles of the first prism and second prism separately.

Fig. 3(a) shows the single wedge prism, where θw is the vertex angle. θd is the deviation angle. The relationship between them is as follows[7],

⎥⎦

⎢ ⎤

⎣

⎡

= −

d d

w n

θ

θ θ

cos

arctan sin (1)

where n is the refractive index of the prism. When a pair of prisms was used, the beam can be steered in any direction in a cone by rotation of the two wedge prisms, as shown in Fig. 3(b). The maximum deviation angle is 2θd. For any point p in XY plane inside the circle, [7] gives the equations to decide the rotation angles of each prism. Although wedge prisms

Proc. of SPIE Vol. 6376 637608-3

can change the view angle, the field of view also changes during rotating the prism. Therefore the scanning mirror was used to move the field of view. The operation includes two steps as shown in Fig. 4, where S is the location of the object.

The first step is to rotate the pair of the wedge prisms to get desired angle as shown in Fig. 4(a). In the second step, scanning mirror will move the filed of view to the object as shown in Fig. 4(b).

θd

2 Y

X p(p_x,p_y)

z

Wedge 1 Wedge 2

(a) (b) Fig. 3. Concept of the wedge prism: (a) single wedge prism and (b) double wedge prisms

Y

S(S_x,SX_y) Wedge 1 Wedge 2

Y

S(S_x,SX_y) Wedge 1 Wedge 2

(a) (b) Fig. 4. Operation of the active vision system: (a) the first step and (b) the second step

2.2 Aberration correction using deformable mirror

However wedge prisms introduce aberrations when they are used in a convergent light, which can degrade the quality of the image. [8] presents some ways to eliminate the aberration. But prisms should be specially positioned. It is not suitable for the proposed system where prisms should rotate separately. In this paper, a 37-channel micromachined membrane deformable mirror device (OKO Technologies) was used in the proposed system. It consists of an aluminum-coated silicon nitride membrane with 37 electrostatic electrodes, which can generate the electrostatic forces between the membrane and electrodes. The wavefront correction is achieved by adjusting the mirror surface to the same shape as the wavefront, but of half the magnitude as shown in Fig. 5. According to [9], the MMDM can compensate the aberration up to 3.5λ (wavelength λ=0.685 µm) and 2λ for lower modes and high order Zernike polynomials respectively.

Incoming wavefront output

wavefront

Deformable mirror Incoming

wavefront output

wavefront

Deformable mirror

Fig.5. Wave-front error correction using deformable mirror θd

θw

Proc. of SPIE Vol. 6376 637608-4

2.3 Concept of deformable mirror

The response of the deformable mirror to a voltage applied can be given by the actuator influence function as follows,

s=Ax

where s is the column Zernike coefficient vector (n×1) of the mirror surface. x is the vector (m×1) of the control signal of the mirror actuators. A is n×m influence matrix. The element aij of A is the slop of linear function between ith coefficient si and the control signal xj of the jth channel, which can be determined by experiment. [10]

So given a desired surface of the mirror, the required voltages can be calculated using following equation.

x=A

⁺

s

where A⁺ is the pseudoinverse of the influence matrix which can be obtained by Singular Value Decomposition method.

In order to measure the wavefront, Hartmann and Hartmann-Shack sensor need stable point source on the object to measure the wavefront. However the reflectance of the micro object is not often high enough to reflect the laser light.

Similar to [11], we also use focus measure information from the camera as the metric to represent the aberration. In this paper, the frequency selective weighted median (FSWM) filter is used to calculate the focus measure. [12]

Owing to the orthogonality of the Zernike polynomials, the proposed deformable mirror control algorithm corrects the aberration in each Zernike mode instead of controlling each actuator. The control algorithm is as follow.

1) The step of control signal δx_i^m is calculated by the equation as follows.

m i m

i

A Z

x δ

δ =

⁺

Where δZ_i^m is the step of Zernike coefficient at mth iteration. i is the index of the Zernike coefficient that we want to correct. A is the inverse of the influence matrix. ⁺

2) The control signal

x

_i^m₊ and

x

_i^m₋ are applied on the deformable mirror separately, which are defined as follow

m i m i m

i

x x

x

₊

=

⁻¹

+ δ , x

_i^m₋

= x

_i^m−1

− δ x

_i^m

We can get the focus measure I+ and I- from the camera.

3) The updated control signal is shown as follows,

m i m m m

i m

i

x I I x

x

⁺¹

= + µ (

₊

−

₋

) ⋅ δ

Where µ is constant. And focus measure I^m+1can be calculate when

x

_i^m+1 is applied on the deformable mirror.

4) If I ^m+1 > I- and I ^m+1 > I+ , the step should decrease as

m

Z

i

δ = δ Z

_i^m

ω

where

ω

is a decrease index. And then go to step 1).

Proc. of SPIE Vol. 6376 637608-5

5) If the difference between I ^m+1 and I ^m-1 are less than a threshold Ic. The control program will correct the aberration in next term of Zernike coefficient repeating from step 1 to 4, until all the Zernike coefficients had been tested.

3. ACTIVE VISION SYSTEM DESIGN AND SIMULATION

3.1 Design process

In this paper, ZEMAX was used to facilitate the optical system design process. The design process consists of three steps. In the first step, the active vision system was divided to imaging optics subsystem and steering optics subsystem.

The selection and layout of the component are made based on the specifications of the proposed system shown in Table 1. The proposed system applied post-objective scanning, which eliminates the design of the scanning lens group. The curved focal plane introduced by the post-objective scanning can be compensated by the deformable mirror. All components of the optical system are off the shelf.

In the second step, the location of each optics component is decided by ZEMAX optimization algorithm. The aberration of the system will be kept inside of 3.5λ (wavelength λ=0.660 µm) by using optimization algorithm.

In the last step, different image analysis tools in ZEMAX were used to evaluate the quality of the system. If the quality of the system is not satisfied, step 1 and 2 will be repeated.

Table 1. Specification of the proposed system

Image space NA: 0.033

Magnification: 1 Field of view: 8mm (diameter) Optical resolution 11.85µm Primary wavelength: 660nm

Effective picture elements: 782 (H) x 582 (V) Cell size: 8.3 um (H) x 8.3 um (V) System Aperture 10mm

Deviation angle (maximum) 20˚

3.2 Simulated result

The final design of active vision system shown in Fig. 6 was developed by repeating the three design steps discussed above. Four achromatic lenses were used, which are chosen from the lens catalog supplied by Edmund Optics. The nominal deviation of the wedge prism used in the proposed system is 10 degree. Therefore the maximum deviation of the pair of wedge prisms is 20 degree. In case of the custom manufactured wedge prism, the larger deviation angle can be achieved, which will enlarge the range of view direction angle. In the simulation, the surface type of the deformable mirror is defined as Zernike standard phase, which is the same as the definition of the aberration in Zemax. In order to get the optimal shape of the deformable mirror, the surface of the deformable mirror was defined as the Zernike standard coefficient of aberration at first. Then the optimization tool in Zemax was used to get the final surface. In this way, optimization process would be fast. Table 2 shows the six configurations defined in Multi-Configuration tools of Zemax.

Table 3 compares the pick-valley wavefront error before correction and after correction of the deformable, which show the ability of deformable mirror to correct the aberration. Table 4 shows the shape of the deformable mirror.

Proc. of SPIE Vol. 6376 637608-6

object plane wedge prism Scanning mirror

deformable mirror

1^stImage plane

system aperture

1^stlens 2^stlens

3^stlens

4^stlens

final image

100 mm 100 mm

Fig. 6. Active vision system design Table 2. Multi configurations

Table 3. Pick-Valley wavefront error (wavelength λ=660nm) θs

θr1 θr2 45.9˚ 45˚ 43.1˚

0 0

0 180

θ_s

45.9˚ 45˚ 43.1˚

θr1 θr2

Uncorrected Corrected Uncorrected Corrected Uncorrected Corrected 0 0 0.4921λ 0.1528λ 0.8951λ 0.1521λ 2.1437λ 0.1221λ 0 180 1.9259λ 0.1423λ 2.0814λ 0.1465λ 2.3469λ 0.1571λ

Proc. of SPIE Vol. 6376 637608-7

F

1

Table 4. Shape of the deformable mirror ( wavelength λ=660nm)

4. EXPERIMENT RESULTS

4.1 Experiment setup

In order to investigate the feasibility of the proposed optical system design, we build up the preliminary experiment setup as shown in Fig. 7. The deformable mirror was supplied by OKO Technologies. The scanning mirror was supplied by General Scanning INC. The deviation angle of wedge prisms is 10°. Because the preliminary experiment setup evaluates the optical performance of the proposed system, the wedge prisms are installed on the manual rotary stages.

1^stlens

2^ndlens 3^rdlens

4^th lens

Iris camera

scanning mirror

wedge prism

deformable mirror

Fig. 7. Experiment setup

In the experiment, an USAF1591 calibration target was used as test target, where group 5 and 4 will be tested. Location of the test area in the image of full field of view is shown in Fig.8.

θs

θr1 θr2

45.9˚ 45˚ 43.1˚

0 0 ^{Weave length}

-1 0

1 -1

0 -0.21

0 0.2

-1 0

1 -1

0 -0.51

0 0.5 Weave length

-1 0

1 -1

0 -11

0 1 2 Weave length

0 180

-1 0

1 -1

0 -21 -1 0 1 Weave length

-1 0

1 -1

0 -21 -1 0 1 Weave length

Weave length

-1 0

1 -1

0 -21 -1 0 1

Proc. of SPIE Vol. 6376 637608-8

2 2E111

3E111

4 ElIl

5 ElIl 6 ElIl

3

.IIIE III

IIIE1 2

test area

Fig. 8. Image of USAF1591 calibration target in the field of view

4.2 Correcting defocus using deformable mirror

The post-objective scanning configuration used in the proposed system will make curved scan filed. And observation of moving micro object also needs to correct the defocus dynamically because of the small depth of field of microscopy.

Therefore the ability of the deformable mirror to correct defocus problem was evaluated without wedge prism. The object plane was shifted from the focused object plane by a distance δZ. In following experiment, group 5 and 4 will be tested. The convergence of focus measure were shown in Fig. 9. As can be seen, when the shifted distance δZ increase, the convergence time will increase while the final focus measure will decrease. The images captured from camera before correction and after correction are shown in Table 5 for three cases. The bars of group 5,element 2 on the target can be resolved at the first two case, which correspond to 36lp/mm. So the resolving power of the preliminary system is about 27um. As can be seen, when the δZ is 8mm, the aberration is too large to be corrected by deformable mirror.

10 20 30 40 50 60 70

200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400

Focus measure

Iteration number

δZ=1mm δZ=2mm δZ=3mm δZ=4mm δZ=5mm δZ=6mm δZ=7mm δZ=8mm

Fig. 9. Focus measure change under different shift distance δZ

Proc. of SPIE Vol. 6376 637608-9

• IN a 5

_U.

hUE4

HI

Ill

HI

HI IHI I

—

Table 5. Captured images before correction and after correction under different shift distance δZ

δZ Before correction After correction

2mm

7mm

8mm

4.3 Correcting aberration introduced by wedge prism using deformable mirror

When wedge prisms were added to the optical path, other aberration such as coma and astigmatism will be introduced to the system. In order to test the ability of the proposed system to correct those aberration, the experiments were made in six configurations as shown in Table 2. Table 6 shows the image of the target without correction of aberration. Table 7 shows the result when those aberrations are corrected by the deformable mirror. As can be seen, the image quality improves when the deformable mirror is used.

Table 6 Captured images before correction

θ_s θr1 θr2

45.9˚ 45˚ 43.1˚

0 0

0 180

Proc. of SPIE Vol. 6376 637608-10

III III • I = =

— — — —

— iii

'II —

'fill' III • = =

— —

£ III

III HiU

—i

144

iiii 9 ^{•'. • 91A III -a}

=uI

4 ^{• FPP IS}

uuit

Table 7. Captured images after correction.

4.4 Changing view direction

The ability of the proposed system to change the view without moving the object is shown in Table 8. The first configuration in Table 8 shows the image of chips with a top view where only top surface was observed. By changing the configuration of the prism and scanning mirror, we can observe the leg of the chip on each side as shown in the second and third configurations in Table 8.

Table 8 Captured images with different view

Configurations

(θs =44.5 , θr1=0°, θr2=180° ) (θs = 43.5, θr1=0°, θr2=0°) (θs =46.5 , θr1=180°, θr2=180°)

Captured images

θs

θr1 θr2 45.9˚ 45˚ 43.1˚

0 0

0 180

Proc. of SPIE Vol. 6376 637608-11

5. CONCLUSIONS

In this paper, we proposed the active optical system that can change the view during observation of the micro object was proposed. It also has the abilities to make autofocus and correct aberration. We also proposed the control algorithm for the deformable mirror. Owing to the orthogonality of the Zernike polynomials, the proposed algorithm can correct the aberration in each Zernike mode instead of controlling each actuator, which simplifies the control issue of deformable mirror. Initial experiments were demonstrated on the preliminary experiment setup to investigate the validity of the concept of the proposed system. Future works will involve improvement of initial design, enhancement of the illumination system and automation of the proposed system.

REFERENCES

1. S. J. Ralis, B. Vikaramadiya, B.J. Nelson, Micropositioning of a Weakly Calibrated Microassembly System Using Coarse-to-Fine Visual Servoing Strategies, IEEE Transactions on Electronics Packaging Manufacturing, Vol 23, No. 2, pp. 123-131, 2000

2. B. Vikramaditya, J. G. Lor, B. J. Nelson, Visually Servoed Micropositioning for Assembly of Hybrid MEMS:

Theory and Experiments, Proceedings of the 36th Conference on Design & Control, p. 96-101,1997

3. J. T. Feddema, R. W. Smon, CAD-Driven Microassembly and Visual Servoing, IEEE International Conference on Robotics & Automation, Leuven, Beligum, pp. 1212-1219, 1998

4. B. Nelson, P. K. Khosla, Integrating Sensor Placement and Visual Tracking Strategies, IEEE International Conference on Robotics and Automation, 1994, pp.1351-1356

5. X. Tao, H.S. Cho and Y. Cho, Microassembly of peg and hole using active zooming, SPIE International Symposium on Optomechatronic Technologies (ISOT 2005), Sapporo, Japan, 2005

6. B. Potsaid, Y. Bellouard, and J. Wen, Adaptive Scanning Optical Microscope (ASOM): A multidisciplinary optical microscope design for large field of view and high resolution imaging, Opt. Express 13, 6504-6518, 2005

7. E. Kobayashi, I. Sakuma, K. Konishi, M. Hashizume, T. Dohi, A robotic wide-angle view endoscope using wedge prisms, Surgical endoscopy, v.18 no.9, 2004, pp.1396-1398

8. J. W. Howard,Formulas for the coma and astigmatism of wedge prisms used in converging light, Appl. Opt. 24, 4265-, 1985

9. L. Zhu, P. C. Sun, D. U. Bartsch, W. R. Freeman, and Y. Fainman, Wave-front generation of Zernike polynomial modes with a micromachined membrane deformable mirror , Appl. Opt. 38, 6019-6026, 1999

10. L. Zhu , P. C. Sun , D. U. Bartsch , W. R. Freeman , and Y. Fainman , Adaptive control of a micromachined continuous membrane deformable mirror for aberration compensation , Appl. Opt. 38 , 168-176, 1999

11. B. Potsaid, J. Wen, and Y. Bellouard, Adaptive scanning optical microscope (ASOM) for large workspace micro-robotic applications, ICRA 2006, 2006, pp.1024-1029

12. K.S. Choi, J. S. Lee, S.J. Ko, New autofocusing technique using the frequency selective weighted median filter for video cameras, IEEE Transactions on Consumer Electronics, Volume 45, Issue 3,pp 820 – 827, Aug., 1999

Proc. of SPIE Vol. 6376 637608-12