Learning Distinct Features Helps, Provably

Firas Laakom, Jenni Raitoharju, Alexandros Iosifidis, Moncef Gabbouj

Research output: Chapter in Book/Report/Conference proceedingConference contributionScientificpeer-review

Abstract

We study the diversity of the features learned by a two-layer neural network trained with the least squares loss. We measure the diversity by the average -distance between the hidden-layer features and theoretically investigate how learning non-redundant distinct features affects the performance of the network. To do so, we derive novel generalization bounds depending on feature diversity based on Rademacher complexity for such networks. Our analysis proves that more distinct features at the network’s units within the hidden layer lead to better generalization. We also show how to extend our results to deeper networks and different losses.

Original languageEnglish
Title of host publicationMachine Learning and Knowledge Discovery in Databases
Subtitle of host publicationResearch Track - European Conference, ECML PKDD 2023, Proceedings
EditorsDanai Koutra, Claudia Plant, Manuel Gomez Rodriguez, Elena Baralis, Francesco Bonchi
PublisherSpringer
Pages206-222
Number of pages17
ISBN (Electronic)978-3-031-43415-0
ISBN (Print)9783031434143
DOIs
Publication statusPublished - 2023
Publication typeA4 Article in conference proceedings
EventEuropean Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases - Turin, Italy
Duration: 18 Sept 202322 Sept 2023

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume14170 LNAI
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

ConferenceEuropean Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases
Country/TerritoryItaly
CityTurin
Period18/09/2322/09/23

Keywords

  • Feature Diversity
  • Generalization Theory
  • Neural Networks

Publication forum classification

  • Publication forum level 1

ASJC Scopus subject areas

  • Theoretical Computer Science
  • General Computer Science

Fingerprint

Dive into the research topics of 'Learning Distinct Features Helps, Provably'. Together they form a unique fingerprint.

Cite this