Constrained Imitation Q-learning with Earth Mover’s Distance reward

Wenyan Yang; Nataliya Strokina; Joni Pajarinen; Joni Kämäräinen

Constrained Imitation Q-learning with Earth Mover’s Distance reward

Computing Sciences

Research output: Other conference contribution › Poster › Scientific

Abstract

We propose constrained Earth Mover’s Distance (CEMD) Imitation Q-learning
that combines the exploration power of Reinforcement Learning (RL) and the
sample efficiency of Imitation Learning (IL). Sample efficiency makes Imitation
Q-learning a suitable approach for robot learning. For Q-learning, immediate
rewards can be efficiently computed by a greedy variant of Earth Mover’s Distance
(EMD) between the observed state-action pairs and state-actions in stored expert
demonstrations. In CEMD, we constrain the otherwise non-stationary greedy EMD
reward by proposing a greedy EMD upper bound estimate and a generic Q-learning
lower bound. In PyBullet continuous control benchmarks, CEMD is more sample
efficient, achieves higher performance and yields less variance than its competitors.

Original language	English
Publication status	Published - 2022
Publication type	Not Eligible
Event	36th Conference on Neural Information Processing Systems - Deep Reinforcement Learning Workshop - New Orleans Ernest N. Morial Convention Center, New Orleans, United States Duration: 28 Nov 2022 → 9 Dec 2022 https://nips.cc/Conferences/2022/ScheduleMultitrack?event=49989

Workshop

Workshop	36th Conference on Neural Information Processing Systems - Deep Reinforcement Learning Workshop
Abbreviated title	NeurIPS 2022 Deep RL workshop
Country/Territory	United States
City	New Orleans
Period	28/11/22 → 9/12/22
Internet address	https://nips.cc/Conferences/2022/ScheduleMultitrack?event=49989

Keywords

Imitation learning
reinforcement learning
Wasserstein distance
Q-learning

Cite this

@conference{06c65bea34f748109a080eee60e14ab1,

title = "Constrained Imitation Q-learning with Earth Mover{\textquoteright}s Distance reward",

abstract = "We propose constrained Earth Mover{\textquoteright}s Distance (CEMD) Imitation Q-learning that combines the exploration power of Reinforcement Learning (RL) and the sample efficiency of Imitation Learning (IL). Sample efficiency makes Imitation Q-learning a suitable approach for robot learning. For Q-learning, immediate rewards can be efficiently computed by a greedy variant of Earth Mover{\textquoteright}s Distance (EMD) between the observed state-action pairs and state-actions in stored expert demonstrations. In CEMD, we constrain the otherwise non-stationary greedy EMD reward by proposing a greedy EMD upper bound estimate and a generic Q-learning lower bound. In PyBullet continuous control benchmarks, CEMD is more sample efficient, achieves higher performance and yields less variance than its competitors.",

keywords = "Imitation learning, reinforcement learning, Wasserstein distance, Q-learning",

author = "Wenyan Yang and Nataliya Strokina and Joni Pajarinen and Joni K{\"a}m{\"a}r{\"a}inen",

year = "2022",

language = "English",

note = "36th Conference on Neural Information Processing Systems - Deep Reinforcement Learning Workshop<br/>, NeurIPS 2022 Deep RL workshop ; Conference date: 28-11-2022 Through 09-12-2022",

url = "https://nips.cc/Conferences/2022/ScheduleMultitrack?event=49989",

}

TY - CONF

T1 - Constrained Imitation Q-learning with Earth Mover’s Distance reward

AU - Yang, Wenyan

AU - Strokina, Nataliya

AU - Pajarinen, Joni

AU - Kämäräinen, Joni

PY - 2022

Y1 - 2022

N2 - We propose constrained Earth Mover’s Distance (CEMD) Imitation Q-learning that combines the exploration power of Reinforcement Learning (RL) and the sample efficiency of Imitation Learning (IL). Sample efficiency makes Imitation Q-learning a suitable approach for robot learning. For Q-learning, immediate rewards can be efficiently computed by a greedy variant of Earth Mover’s Distance (EMD) between the observed state-action pairs and state-actions in stored expert demonstrations. In CEMD, we constrain the otherwise non-stationary greedy EMD reward by proposing a greedy EMD upper bound estimate and a generic Q-learning lower bound. In PyBullet continuous control benchmarks, CEMD is more sample efficient, achieves higher performance and yields less variance than its competitors.

AB - We propose constrained Earth Mover’s Distance (CEMD) Imitation Q-learning that combines the exploration power of Reinforcement Learning (RL) and the sample efficiency of Imitation Learning (IL). Sample efficiency makes Imitation Q-learning a suitable approach for robot learning. For Q-learning, immediate rewards can be efficiently computed by a greedy variant of Earth Mover’s Distance (EMD) between the observed state-action pairs and state-actions in stored expert demonstrations. In CEMD, we constrain the otherwise non-stationary greedy EMD reward by proposing a greedy EMD upper bound estimate and a generic Q-learning lower bound. In PyBullet continuous control benchmarks, CEMD is more sample efficient, achieves higher performance and yields less variance than its competitors.

KW - Imitation learning

KW - reinforcement learning

KW - Wasserstein distance

KW - Q-learning

M3 - Poster

T2 - 36th Conference on Neural Information Processing Systems - Deep Reinforcement Learning Workshop<br/>

Y2 - 28 November 2022 through 9 December 2022

ER -

Constrained Imitation Q-learning with Earth Mover’s Distance reward

Abstract

Workshop

Keywords

Fingerprint

Cite this