LiDAR Place Recognition Evaluation with the Oxford Radar RobotCar Dataset Revised

Jukka Peltomäki; Farid Alijani; Jussi Puura; Heikki Huttunen; Esa Rahtu; Joni Kristian Kämäräinen

doi:10.1007/978-3-031-31435-3_1

LiDAR Place Recognition Evaluation with the Oxford Radar RobotCar Dataset Revised

Jukka Peltomäki, Farid Alijani, Jussi Puura, Heikki Huttunen, Esa Rahtu, Joni Kristian Kämäräinen

Tietotekniikka

Tutkimustuotos: Konferenssiartikkeli › Tieteellinen › vertaisarvioitu

3 Lataukset (Pure)

Abstrakti

The Oxford Radar RobotCar dataset has recently become popular in evaluating LiDAR-based methods for place recognition. The Radar dataset is preferred over the original Oxford RobotCar dataset since it has better LiDAR sensors and location ground truth is available for all sequences. However, it turns out that the Radar dataset has serious issues with its ground truth and therefore experimental findings with this dataset can be misleading. We demonstrate how easily this can happen, by varying only the gallery sequence and keeping the training and test sequences fixed. Results of this experiment strongly indicate that the gallery selection is an important consideration for place recognition. However, the finding is a mistake and the difference between galleries can be explained by systematic errors in the ground truth. In this work, we propose a revised benchmark for LiDAR-based place recognition with the Oxford Radar RobotCar dataset. The benchmark includes fixed gallery, training and test sequences, corrected ground truth, and a strong baseline method. All data and code will be made publicly available to facilitate fair method comparison and development.

Alkuperäiskieli	Englanti
Otsikko	Image Analysis - 23rd Scandinavian Conference, SCIA 2023, Proceedings
Toimittajat	Rikke Gade, Michael Felsberg, Joni-Kristian Kämäräinen
Kustantaja	Springer
Sivut	3-16
Sivumäärä	14
ISBN (painettu)	9783031314346
DOI - pysyväislinkit	https://doi.org/10.1007/978-3-031-31435-3_1
Tila	Julkaistu - 2023
OKM-julkaisutyyppi	A4 Artikkeli konferenssijulkaisussa
Tapahtuma	Scandinavian Conference on Image Analysis - Lapland, Suomi Kesto: 18 huhtik. 2023 → 21 huhtik. 2023

Julkaisusarja

Nimi	Lecture Notes in Computer Science
Vuosikerta	13885 LNCS
ISSN (painettu)	0302-9743
ISSN (elektroninen)	1611-3349

Conference

Conference	Scandinavian Conference on Image Analysis
Maa/Alue	Suomi
Kaupunki	Lapland
Ajanjakso	18/04/23 → 21/04/23

Julkaisufoorumi-taso

Jufo-taso 1

!!ASJC Scopus subject areas

Theoretical Computer Science
Yleinen tietojenkäsittelytiede

Pääsy asiakirjaan

10.1007/978-3-031-31435-3_1

LiDAR_Place_Recognition_Evaluation_with_the_Oxford_Radar_RobotCar_Dataset_Revised__for_SCIA2023_
This publication is copyrighted. You may download, display and print it for Your own personal use. Commercial use is prohibited.
Accepted author manuscript, 4,68 MBLisenssi: Unspecified

https://urn.fi/URN:NBN:fi:tuni-202406197308Lisenssi: Unspecified

Siteeraa tätä

Peltomäki, J., Alijani, F., Puura, J., Huttunen, H., Rahtu, E., & Kämäräinen, J. K. (2023). LiDAR Place Recognition Evaluation with the Oxford Radar RobotCar Dataset Revised. teoksessa R. Gade, M. Felsberg, & J.-K. Kämäräinen (Toimittajat), Image Analysis - 23rd Scandinavian Conference, SCIA 2023, Proceedings (Sivut 3-16). (Lecture Notes in Computer Science; Vuosikerta 13885 LNCS). Springer. https://doi.org/10.1007/978-3-031-31435-3_1

@inproceedings{83b597fa11c14d909f9dc878d85b252f,

title = "LiDAR Place Recognition Evaluation with the Oxford Radar RobotCar Dataset Revised",

abstract = "The Oxford Radar RobotCar dataset has recently become popular in evaluating LiDAR-based methods for place recognition. The Radar dataset is preferred over the original Oxford RobotCar dataset since it has better LiDAR sensors and location ground truth is available for all sequences. However, it turns out that the Radar dataset has serious issues with its ground truth and therefore experimental findings with this dataset can be misleading. We demonstrate how easily this can happen, by varying only the gallery sequence and keeping the training and test sequences fixed. Results of this experiment strongly indicate that the gallery selection is an important consideration for place recognition. However, the finding is a mistake and the difference between galleries can be explained by systematic errors in the ground truth. In this work, we propose a revised benchmark for LiDAR-based place recognition with the Oxford Radar RobotCar dataset. The benchmark includes fixed gallery, training and test sequences, corrected ground truth, and a strong baseline method. All data and code will be made publicly available to facilitate fair method comparison and development.",

keywords = "evaluation protocol, ground truth position accuracy, Oxford Radar RobotCar dataset, place recognition",

author = "Jukka Peltom{\"a}ki and Farid Alijani and Jussi Puura and Heikki Huttunen and Esa Rahtu and K{\"a}m{\"a}r{\"a}inen, {Joni Kristian}",

note = "Publisher Copyright: {\textcopyright} 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.; Scandinavian Conference on Image Analysis ; Conference date: 18-04-2023 Through 21-04-2023",

year = "2023",

doi = "10.1007/978-3-031-31435-3_1",

language = "English",

isbn = "9783031314346",

series = "Lecture Notes in Computer Science",

publisher = "Springer",

pages = "3--16",

editor = "Rikke Gade and Michael Felsberg and Joni-Kristian K{\"a}m{\"a}r{\"a}inen",

booktitle = "Image Analysis - 23rd Scandinavian Conference, SCIA 2023, Proceedings",

}

Peltomäki, J, Alijani, F, Puura, J, Huttunen, H, Rahtu, E & Kämäräinen, JK 2023, LiDAR Place Recognition Evaluation with the Oxford Radar RobotCar Dataset Revised. julkaisussa R Gade, M Felsberg & J-K Kämäräinen (toim), Image Analysis - 23rd Scandinavian Conference, SCIA 2023, Proceedings. Lecture Notes in Computer Science, Vuosikerta 13885 LNCS, Springer, Sivut 3-16, Scandinavian Conference on Image Analysis, Lapland, Suomi, 18/04/23. https://doi.org/10.1007/978-3-031-31435-3_1

LiDAR Place Recognition Evaluation with the Oxford Radar RobotCar Dataset Revised. / Peltomäki, Jukka; Alijani, Farid; Puura, Jussi et al.
Image Analysis - 23rd Scandinavian Conference, SCIA 2023, Proceedings. toim. / Rikke Gade; Michael Felsberg; Joni-Kristian Kämäräinen. Springer, 2023. s. 3-16 (Lecture Notes in Computer Science; Vuosikerta 13885 LNCS).

Tutkimustuotos: Konferenssiartikkeli › Tieteellinen › vertaisarvioitu

TY - GEN

T1 - LiDAR Place Recognition Evaluation with the Oxford Radar RobotCar Dataset Revised

AU - Peltomäki, Jukka

AU - Alijani, Farid

AU - Puura, Jussi

AU - Huttunen, Heikki

AU - Rahtu, Esa

AU - Kämäräinen, Joni Kristian

PY - 2023

Y1 - 2023

N2 - The Oxford Radar RobotCar dataset has recently become popular in evaluating LiDAR-based methods for place recognition. The Radar dataset is preferred over the original Oxford RobotCar dataset since it has better LiDAR sensors and location ground truth is available for all sequences. However, it turns out that the Radar dataset has serious issues with its ground truth and therefore experimental findings with this dataset can be misleading. We demonstrate how easily this can happen, by varying only the gallery sequence and keeping the training and test sequences fixed. Results of this experiment strongly indicate that the gallery selection is an important consideration for place recognition. However, the finding is a mistake and the difference between galleries can be explained by systematic errors in the ground truth. In this work, we propose a revised benchmark for LiDAR-based place recognition with the Oxford Radar RobotCar dataset. The benchmark includes fixed gallery, training and test sequences, corrected ground truth, and a strong baseline method. All data and code will be made publicly available to facilitate fair method comparison and development.

AB - The Oxford Radar RobotCar dataset has recently become popular in evaluating LiDAR-based methods for place recognition. The Radar dataset is preferred over the original Oxford RobotCar dataset since it has better LiDAR sensors and location ground truth is available for all sequences. However, it turns out that the Radar dataset has serious issues with its ground truth and therefore experimental findings with this dataset can be misleading. We demonstrate how easily this can happen, by varying only the gallery sequence and keeping the training and test sequences fixed. Results of this experiment strongly indicate that the gallery selection is an important consideration for place recognition. However, the finding is a mistake and the difference between galleries can be explained by systematic errors in the ground truth. In this work, we propose a revised benchmark for LiDAR-based place recognition with the Oxford Radar RobotCar dataset. The benchmark includes fixed gallery, training and test sequences, corrected ground truth, and a strong baseline method. All data and code will be made publicly available to facilitate fair method comparison and development.

KW - evaluation protocol

KW - ground truth position accuracy

KW - Oxford Radar RobotCar dataset

KW - place recognition

U2 - 10.1007/978-3-031-31435-3_1

DO - 10.1007/978-3-031-31435-3_1

M3 - Conference contribution

AN - SCOPUS:85161430611

SN - 9783031314346

T3 - Lecture Notes in Computer Science

SP - 3

EP - 16

BT - Image Analysis - 23rd Scandinavian Conference, SCIA 2023, Proceedings

A2 - Gade, Rikke

A2 - Felsberg, Michael

A2 - Kämäräinen, Joni-Kristian

PB - Springer

T2 - Scandinavian Conference on Image Analysis

Y2 - 18 April 2023 through 21 April 2023

ER -

Peltomäki J, Alijani F, Puura J, Huttunen H, Rahtu E , Kämäräinen JK. LiDAR Place Recognition Evaluation with the Oxford Radar RobotCar Dataset Revised. julkaisussa Gade R, Felsberg M, Kämäräinen JK, toimittajat, Image Analysis - 23rd Scandinavian Conference, SCIA 2023, Proceedings. Springer. 2023. s. 3-16. (Lecture Notes in Computer Science). doi: 10.1007/978-3-031-31435-3_1