Journal of the Korean Society of Surveying, Geodesy, Photogrammetry and Cartography Vol. 37, No. 5, 331-341, 2019
https://doi.org/10.7848/ksgpc.2019.37.5.331
Updating Smartphone’s Exterior Orientation Parameters by Image-based Localization Method Using Geo-tagged Image
Datasets and 3D Point Cloud as References
Wang, Ying Hsuan1)·Hong, Seunghwan2)·Bae, Junsu3)·Choi, Yoonjo4)·Sohn, Hong-Gyoo5)
Abstract
With the popularity of sensor-rich environments, smartphones have become one of the major platforms for obtaining and sharing information. Since it is difficult to utilize GNSS (Global Navigation Satellite System) inside the area with many buildings, the localization of smartphone in this case is considered as a challenging task. To resolve problem of localization using smartphone a four step image-based localization method and procedure is proposed. To improve the localization accuracy of smartphone datasets, MMS (Mobile Mapping System) and Google Street View were utilized. In our approach first, the searching for candidate matching image is performed by the query image of smartphone’s using GNSS observation. Second, the SURF (Speed-Up Robust Features) image matching between the smartphone image and reference dataset is done and the wrong matching points are eliminated. Third, the geometric transformation is performed using the matching points with 2D affine transformation. Finally, the smartphone location and attitude estimation are done by PnP (Perspective- n-Point) algorithm. The location of smartphone GNSS observation is improved from the original 10.204m to a mean error of 3.575m. The attitude estimation is lower than 25 degrees from the 92.4% of the adjsuted images with an average of 5.1973 degrees.
Keywords : Smartphone, Image-based Localization, MMS (Mobile Mapping System), Google Street View
ISSN 1598-4850(Print) ISSN 2288-260X(Online) Original article
1. Introduction
Recently, smartphones have become an important platform for sharing information. Current smartphones are equipped with high resolution cameras, IMU (Inertial Measurement Unit), GNSS (Global Navigation Satellite System), and other sensors with new releases. Since the wide availability of sensor-rich environments, smartphone changes not
only the way people communicate but also the way people interacts with the society. Mobile crowdsensing technology that uses ubiquitous mobile devices to collect information about human activity and surrounding environment from integrated sensors on a mobile device has been growing.
Information can be collected in the format of images, texts, and videos by smartphones. Moreover, the smartphone with GNSS and IMU sensors can offer location information
Received 2019. 09. 24, Revised 2019. 10. 07, Accepted 2019. 10. 11
1) Dept. of Civil and Environmental Engineering, Yonsei University (E-mail: [email protected]) 2) Member, Stryx Inc. (E-mail: [email protected])
3) Member, Dept. of Civil and Environmental Engineering, Yonsei University (E-mail: [email protected]) 4) Member, Dept. of Civil and Environmental Engineering, Yonsei University (E-mail: [email protected])
5) Corresponding Author, Member, Dept. of Civil and Environmental Engineering, Yonsei University (E-mail: [email protected])
This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://
creativecommons.org/licenses/by-nc/3.0) which permits unrestricted non-commercial use, distribution, and reproduction in any medium,
and the user’s perspective of the text, image, and video.
Even though most of the smartphones have been equipped with GNSS receiver, the location accuracy of smartphone is limited especially when GNSS receiver is located between or within urban areas. The overall accuracy of smartphone’s GNSS observation accuracy is about 12.5 m in the outdoor environment (Zandbergen and Barbeau, 2011). Sometimes the inaccurate location and perspective information of smartphone make the information collected by smartphone less useful.
Considering smartphones have been equipped with high- resolution camera, image-based localization becomes an alternative solution to correct the location and attitude of the smartphone. Image-based localization has been applied in the various fields of study, including robot localization (Agarwal et al., 2015), visual SLAM (Simultaneous Localization And Mapping) (Fuentes-Pacheco et al., 2015), augmented reality (Schönberger et al., 2018), and user localization(Liu et al., 2012; Wu et al., 2018). Image-based localization usually performs in two steps : (1) place recognition and (2) calculate the location and attitude of the camera by PnP (Perspective-n- Point) algorithm. Place recognition, a task of determining the location depicted in a query image by retrieving a given geo- tagged image database, is a challenging task for the success of the image-based localization. In previous researches of place recognition, geo-tagged images have played an important role. The dataset such as INRIA Holidays dataset (Jégou et al., 2008), San Francisco street view image datasets (Sattler et al., 2015; Jiang et al., 2012; Liu et al., 2012; Sattler et al., 2016; Kim et al., 2017), and Google Street View Image datasets were used (Zamir and Shah 2010; Agarwal et al., 2015; Verstockt et al., 2015; Sadeghi, et al., 2016). However, when it comes to image-based localization by street view images, the low spatial distribution and the geopostional accuracy of street view image datasets limit the accuracy of localization results, which is about 12m in the case of Google Street View images (Salarian et al., 2015).
To improve the accuracy of localization of smartphone’s, the utilization of mobile mapping technology is proposed in this study. MMS (Mobile Mapping System) can not only offer street view images in both high resolution and high spatial density but also 3D point cloud information from LiDAR (Light Detection and Ranging). The reference data utilized
in this study is collected by Leica Pegasus: Two Ⅱ including street view of images and 3D point cloud information. Since the IOPs (Interior Orientation Parameters) of the smartphone camera may be different from the specification offered by the manufacturer, the calibration of the smartphone camera is performed before capturing images. The proposed image- based localization is performed in four parts: automatic searching for the candidate image query, image matching between the smartphone image and reference image, geometric transformation estimation, and the improvement of smartphone position and attitude estimation.
To show the potential of applying our proposed image- based localization method to the publically available street view image dataset, Google Street View images are also used as a reference dataset. The structure of the paper is as follows:
Section 2 describes the proposed smartphone image-based localization method. Section 3 describes the experiments and results.
2. Image-based Localization
The proposed image-based localization method uses two different kinds of geo-tagged reference images: one from street view image collected by MMS and the other from Google Street View images. The query images are taken from the smartphone camera. The image-based localization of smartphone is performed in four steps in this study. First, the matching candidate image is queried from the reference image dataset. Second, image matching is done by the SURF (Speed-Up Robust Features) algorithm. Third, a 2D affine transformation is applied for geometric transformation between the smartphone image and reference image. Finally, PnP algorithm is used for refining the smartphone position and attitude estimation. The procedure of the proposed image-based localization method is depicted in Fig. 1.
Fig. 1. Workflow of the proposed image-based localization method
