33-4 / U. Vogel
• IMID 2009 DIGEST
Abstract
First demonstrators of bi-directional OLED microdisplay devices have been developed and integrated into see-through HMD optics. The device combines 'display' and 'imaging' by nested OLED pixels and photodetectors in a single CMOS chip. Major aim of this integration is to provide capabilities for eye-tracking to achieve gaze-based human-display-interaction.
1. Introduction
OLED-on-CMOS microdisplays can be used for near-to-eye (NTE) applications such as viewfinders of cameras or head-mounted displays (HMD). Especially in see-through augmented-reality (AR) applications it would be helpful if the user has a possibility to interact with the display content in an augmented-reality setting while preserving hands-free operation. Such user interaction can be handled by an eye-tracker based on an integrated near-infrared (NIR) camera (gaze control). Therefore integration of NIR camera photodetectors and AMOLED microdisplay into a single device will lead to a highly integrated, very light-weight, small-sized bi-directional microdisplay ("OLEDCam"), which can be used as visual Input-/Output-Device (I/O) for personal information management (PIM). Applications could be in mobile communication (connected to smartphone), industry (worker assistance), medicine (surgeon assistance), security (surveillance, pilots), barrier-free operation (handicapped people), travel/transport (driver assistance, tourist information), or others.
Eyepiece OLEDCam OLG local hardware information server (wireless) Gateway User I/F
Fig. 1. Basic concept of a see-through HMD for interactive augmented-reality applications
2. Experimental approach
The use of OLEDs in active-matrix (AM) CMOS substrates requires a top-emitting, low voltage and highly efficient OLED stack. Bi-directional AMOLED-microdisplays potentially require high-speed OLED switching behaviour (<100µs), since display and eye-tracker camera have to work time-sequentially to avoid optical crosstalk between display and camera, and to minimize distortion in the eye-tracker camera.
An AR system combines the real world with additional computer-generated objects. Several applications like medical, military aircraft, manufacturing and entertainment increasingly use AR systems as a human-computer interaction tool. Common AR systems have low form factor, high power dissipation and are less mobile. Using an OLEDCam device may result in reducing external electrical and optical components and downsizing the HMD. Another aspect to realize a light-weight AR system is a high-performance mobile host-system. This handles the camera images from the eye scene and calculates control information, e.g., the mouse-cursor control.
OLED-Microdisplay with embedded camera for HMD applications
Uwe Vogel, Rigo Herold, Daniel Kreye, Bernd Richter, Gerd Bunk, Sven Reckziegel,
Michael Scholles, Christiane Grillberger, Michael Toerker, Jörg Amelung
1Fraunhofer Institute for Photonic Microsystems, Dresden, D-01109, Germany
Tel.:+49-351-8823-282, E-mail: [email protected]
33-4 / U. Vogel
IMID 2009 DIGEST •
A first prototype of a bi-directional OLED microdisplay device has been designed and manufactured. Major aim of this integration is to provide small form-factor display and eye-tracking for see-through HMD applications (augmented-reality).
3. Results
3.1 Prototype OLEDCam device
Based on OLED pixel and photodiode co-integration an OLEDCam device has been designed. This unique device consists of a nested matrix of OLED pixels and photodiodes, as shown in Fig. 2. The integrated electronic components to drive the OLED microdisplay and read out the camera are illustrated in Fig. 3.
Fig. 2. Nested matrix of OLED pixel and photodiodes
Nestet matrix of OLED-Pixel and photodiodes
OLED-Microdisplay column drivers OL ED -Mi cro dis pla y ro w dri ve rs Digital-to-analog converter OLED-Microdisplay control unit OLED-Microdisplay shift registers Syn. singals Video data Column amplifiers Ca me ra row dri ve r Nestet matrix of OLED-Pixel and photodiodes
OLED-Microdisplay column drivers OL ED -Mi cro dis pla y ro w dri ve rs Digital-to-analog converter OLED-Microdisplay control unit OLED-Microdisplay shift registers Syn. singals Video data Column amplifiers Ca me ra row dri ve r Ca me ra row ad dre ss ing Analog multiplexer Camera column addressing Output amplifier Camera control unit Camera data Synch. signals Control signals Cont. signals
Fig. 3. OLEDCam circuit block diagram 3.2 Bi-directional see-through HMD optics
The bi-directional optics combines the OLEDCam with the optical system, performing different tasks:
First providing see-through functionality from the human eye to the real world. A second function is the projection of a magnified virtual image from the OLEDCam microdisplay. A further capability is capturing the user's point of gaze. The bidirectional optics has to perform all functions at the same time. Because of the entanglement of the optical paths from the OLEDCam surface it is required to decouple the spectral range between the microdisplay and the camera. The microdisplay operates in the visible range (VIS) and the camera in the near-infrared range (NIR). To provide near-infrared light, an NIR illuminator is required on the bidirectional optics (NIR-LED).
accommodation level
spherical mirror for eyetracking eyetracking camera
eye
See-t hrough eyetracking light path
back window
beam split ter
See-t hrough light path
front window
accommodation level
spherical mirror for eyetracking eyetracking camera
eye
See-t hrough eyetracking light path
back window
beam split ter
See-t hrough light path
front window
Fig. 4. Optical paths of the eye-capturing see-through testbox
Fig. 5. Testbox photograph with fully illuminated microdisplay
33-4 / U. Vogel
• IMID 2009 DIGEST
4. Summary
First prototypes of OLED bi-directional microdisplay have been developed and successfully initially operated. Conceptions of initial bi-directional optics and challenges of a first optical test box are discussed. Ongoing work is the design of a full-colour OLEDCam with VGA resolution, integrated camera-readout components, improved bi-directional optics and eye-tracking in an AR system environment (supported by the Fraunhofer Internal Programs under Grant No. WISA 817 805, see www.istar-project.org).
Acknowledgement
This work was partly funded by grants from the Federal Ministry for Education and Research (Bundesministerium für Bildung und Forschung,
BMBF 01 BK 916-919, 16SV2283) of the German government and the Sächsische Aufbaubank (SAB) of the State of Saxony (11107/1733). Further thanks are directed to Unit of Engineering Psychology and Cognitive Ergonomics, Institute for Psychology III, Technische Universität
Dresden (S.-T. Graupner, S. Pannasch, M. Heubner, B. M. Velichkovsky) for cooperation in the eye-tracking field as well as Institut für Angewandte Photophysik, Technische Universität Dresden (K. Leo), and NOVALED AG Dresden (J. Blochwitz-Nimoth). Last but not least the authors like to thank several colleagues and further contributors at Fraunhofer IPMS (e.g., C.
Luber, C. Zschippang, P. König, S. Ulbricht).
5. References
[1] R. Azuma, Y. Baillot, R. Behringer, S. Feiner, S.
Julier, B. MacIntyre: Recent Advances in Augmented Reality, IEEE Computer Graphics and Application,. vol. 21, no. 6 pp. 34-42, 2001.
[2] U. Vogel, D. Kreye, G. Bunk, S. Reckziegel, R.
Herold, M. Scholles, M. Törker, C. Grillberger, J. Amelung: Bi-directional OLED Microdisplay for Interactive HMD, SID 08 DIGEST, San Jose, 2008, pp. 81-84.
[3] J. Duparre and R. Völkel: Novel
Optics/Micro-Optics for Miniature Imaging Systems, Proc. of SPIE, vol. 6196, April 2006.
[4] L. Dongheng: Low-cost eye-tracking for human
computer interaction, Master´s thesis, Iowa State University, 2006.
[5] U. Vogel, D. Kreye, M. Törker, J. Amelung;
Evaluation of RGB OLED Stacks on CMOS
Microdisplay Substrates, IDW ’06 – The 13th International Display Workshops, December 6-8, 2006, Otsu Prince Hotel, Otsu, Japan, pp. 445-448
[6] U. Vogel, D. Kreye, B. Richter, G. Bunk, S.
Reckziegel, R. Herold, M. Scholles, M. Törker, J. Amelung: Bi-directional OLED Microdisplay, IDW ’07 – The 14th International Display Workshops, December 5-7, 2007, Sapporo Convention Center, Sapporo, Japan, pp. 1051-1054
[7] U. Vogel, D. Kreye, B. Richter, G. Bunk, S.
Reckziegel, R. Herold, M. Scholles, M. Törker, C. Grillberger, J. Amelung, S. Graupner, S. Pannasch, M. Heubner, and B. Velichkovsky: Bi-directional OLED microdisplay for interactive see-through HMDs: Study toward integration of eye-tracking and informational facilities, J. Soc. Inf. Display 17, 175 (2009)
