Constrained Imitation Q-learning with Earth Mover’s Distance reward

Wenyan Yang; Nataliya Strokina; Joni Pajarinen; Joni Kämäräinen

Constrained Imitation Q-learning with Earth Mover’s Distance reward

Wenyan Yang, Nataliya Strokina, Joni Pajarinen, Joni Kämäräinen

Tietotekniikka

Tutkimustuotos: Posteri › Scientific

Abstrakti

We propose constrained Earth Mover’s Distance (CEMD) Imitation Q-learning
that combines the exploration power of Reinforcement Learning (RL) and the
sample efficiency of Imitation Learning (IL). Sample efficiency makes Imitation
Q-learning a suitable approach for robot learning. For Q-learning, immediate
rewards can be efficiently computed by a greedy variant of Earth Mover’s Distance
(EMD) between the observed state-action pairs and state-actions in stored expert
demonstrations. In CEMD, we constrain the otherwise non-stationary greedy EMD
reward by proposing a greedy EMD upper bound estimate and a generic Q-learning
lower bound. In PyBullet continuous control benchmarks, CEMD is more sample
efficient, achieves higher performance and yields less variance than its competitors.

Alkuperäiskieli	Englanti
Tila	Julkaistu - 2022
OKM-julkaisutyyppi	Ei OKM-tyyppiä
Tapahtuma	36th Conference on Neural Information Processing Systems - Deep Reinforcement Learning Workshop - New Orleans Ernest N. Morial Convention Center, New Orleans, Yhdysvallat Kesto: 28 marrask. 2022 → 9 jouluk. 2022 https://nips.cc/Conferences/2022/ScheduleMultitrack?event=49989

Workshop

Workshop	36th Conference on Neural Information Processing Systems - Deep Reinforcement Learning Workshop
Lyhennettä	NeurIPS 2022 Deep RL workshop
Maa/Alue	Yhdysvallat
Kaupunki	New Orleans
Ajanjakso	28/11/22 → 9/12/22
www-osoite	https://nips.cc/Conferences/2022/ScheduleMultitrack?event=49989

Siteeraa tätä

@conference{06c65bea34f748109a080eee60e14ab1,

title = "Constrained Imitation Q-learning with Earth Mover{\textquoteright}s Distance reward",

abstract = "We propose constrained Earth Mover{\textquoteright}s Distance (CEMD) Imitation Q-learningthat combines the exploration power of Reinforcement Learning (RL) and thesample efficiency of Imitation Learning (IL). Sample efficiency makes ImitationQ-learning a suitable approach for robot learning. For Q-learning, immediaterewards can be efficiently computed by a greedy variant of Earth Mover{\textquoteright}s Distance(EMD) between the observed state-action pairs and state-actions in stored expertdemonstrations. In CEMD, we constrain the otherwise non-stationary greedy EMDreward by proposing a greedy EMD upper bound estimate and a generic Q-learninglower bound. In PyBullet continuous control benchmarks, CEMD is more sampleefficient, achieves higher performance and yields less variance than its competitors.",

keywords = "Imitation learning, reinforcement learning, Wasserstein distance, Q-learning",

author = "Wenyan Yang and Nataliya Strokina and Joni Pajarinen and Joni K{\"a}m{\"a}r{\"a}inen",

year = "2022",

language = "English",

note = "36th Conference on Neural Information Processing Systems - Deep Reinforcement Learning Workshop<br/>, NeurIPS 2022 Deep RL workshop ; Conference date: 28-11-2022 Through 09-12-2022",

url = "https://nips.cc/Conferences/2022/ScheduleMultitrack?event=49989",

}

TY - CONF

T1 - Constrained Imitation Q-learning with Earth Mover’s Distance reward

AU - Yang, Wenyan

AU - Strokina, Nataliya

AU - Pajarinen, Joni

AU - Kämäräinen, Joni

PY - 2022

Y1 - 2022

N2 - We propose constrained Earth Mover’s Distance (CEMD) Imitation Q-learningthat combines the exploration power of Reinforcement Learning (RL) and thesample efficiency of Imitation Learning (IL). Sample efficiency makes ImitationQ-learning a suitable approach for robot learning. For Q-learning, immediaterewards can be efficiently computed by a greedy variant of Earth Mover’s Distance(EMD) between the observed state-action pairs and state-actions in stored expertdemonstrations. In CEMD, we constrain the otherwise non-stationary greedy EMDreward by proposing a greedy EMD upper bound estimate and a generic Q-learninglower bound. In PyBullet continuous control benchmarks, CEMD is more sampleefficient, achieves higher performance and yields less variance than its competitors.

AB - We propose constrained Earth Mover’s Distance (CEMD) Imitation Q-learningthat combines the exploration power of Reinforcement Learning (RL) and thesample efficiency of Imitation Learning (IL). Sample efficiency makes ImitationQ-learning a suitable approach for robot learning. For Q-learning, immediaterewards can be efficiently computed by a greedy variant of Earth Mover’s Distance(EMD) between the observed state-action pairs and state-actions in stored expertdemonstrations. In CEMD, we constrain the otherwise non-stationary greedy EMDreward by proposing a greedy EMD upper bound estimate and a generic Q-learninglower bound. In PyBullet continuous control benchmarks, CEMD is more sampleefficient, achieves higher performance and yields less variance than its competitors.

KW - Imitation learning

KW - reinforcement learning

KW - Wasserstein distance

KW - Q-learning

M3 - Poster

T2 - 36th Conference on Neural Information Processing Systems - Deep Reinforcement Learning Workshop<br/>

Y2 - 28 November 2022 through 9 December 2022

ER -

Constrained Imitation Q-learning with Earth Mover’s Distance reward

Abstrakti

Workshop

Sormenjälki

Siteeraa tätä