WLD-Reg: A Data-Dependent Within-Layer Diversity Regularizer

Research output: Chapter in Book/Report/Conference proceedingConference contributionScientificpeer-review

Abstract

Neural networks are composed of multiple layers arranged in a hierarchical structure jointly trained with a gradient-based optimization, where the errors are back-propagated from the last layer back to the first one. At each optimization step, neurons at a given layer receive feedback from neurons belonging to higher layers of the hierarchy. In this paper, we propose to complement this traditional’between-layer’ feedback with additional’within-layer’ feedback to encourage the diversity of the activations within the same layer. To this end, we measure the pairwise similarity between the outputs of the neurons and use it to model the layer’s overall diversity. We present an extensive empirical study confirming that the proposed approach enhances the performance of several state-of-the-art neural network models in multiple tasks. The code is publically available at https://github.com/firasl/AAAI-23WLD-Reg.

Original languageEnglish
Title of host publicationAAAI-23 Technical Tracks 7
EditorsBrian Williams, Yiling Chen, Jennifer Neville
Pages8421-8429
Number of pages9
ISBN (Electronic)9781577358800
DOIs
Publication statusPublished - 27 Jun 2023
Publication typeA4 Article in conference proceedings
EventAAAI Conference on Artificial Intelligence - Washington, United States
Duration: 7 Feb 202314 Feb 2023

Publication series

NameProceedings of the 37th AAAI Conference on Artificial Intelligence, AAAI 2023
Number7
Volume37
ISSN (Electronic)2374-3468

Conference

ConferenceAAAI Conference on Artificial Intelligence
Country/TerritoryUnited States
CityWashington
Period7/02/2314/02/23

Publication forum classification

  • Publication forum level 2

ASJC Scopus subject areas

  • Artificial Intelligence

Fingerprint

Dive into the research topics of 'WLD-Reg: A Data-Dependent Within-Layer Diversity Regularizer'. Together they form a unique fingerprint.

Cite this