Learning Distinct Features Helps, Provably

Firas Laakom; Jenni Raitoharju; Alexandros Iosifidis; Moncef Gabbouj

doi:10.1007/978-3-031-43415-0_13

Learning Distinct Features Helps, Provably

Firas Laakom, Jenni Raitoharju, Alexandros Iosifidis, Moncef Gabbouj

Tietotekniikka

Tutkimustuotos: Konferenssiartikkeli › Tieteellinen › vertaisarvioitu

3 Sitaatiot (Scopus)

Abstrakti

We study the diversity of the features learned by a two-layer neural network trained with the least squares loss. We measure the diversity by the average -distance between the hidden-layer features and theoretically investigate how learning non-redundant distinct features affects the performance of the network. To do so, we derive novel generalization bounds depending on feature diversity based on Rademacher complexity for such networks. Our analysis proves that more distinct features at the network’s units within the hidden layer lead to better generalization. We also show how to extend our results to deeper networks and different losses.

Alkuperäiskieli	Englanti
Otsikko	Machine Learning and Knowledge Discovery in Databases
Alaotsikko	Research Track - European Conference, ECML PKDD 2023, Proceedings
Toimittajat	Danai Koutra, Claudia Plant, Manuel Gomez Rodriguez, Elena Baralis, Francesco Bonchi
Kustantaja	Springer
Sivut	206-222
Sivumäärä	17
ISBN (elektroninen)	978-3-031-43415-0
ISBN (painettu)	9783031434143
DOI - pysyväislinkit	https://doi.org/10.1007/978-3-031-43415-0_13
Tila	Julkaistu - 2023
OKM-julkaisutyyppi	A4 Artikkeli konferenssijulkaisussa
Tapahtuma	European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases - Turin, Italia Kesto: 18 syysk. 2023 → 22 syysk. 2023

Julkaisusarja

Nimi	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Vuosikerta	14170 LNAI
ISSN (painettu)	0302-9743
ISSN (elektroninen)	1611-3349

Conference

Conference	European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases
Maa/Alue	Italia
Kaupunki	Turin
Ajanjakso	18/09/23 → 22/09/23

Rahoitus

This work has been supported by the NSF-Business Finland Center for Visual and Decision Informatics (CVDI) project AMALIA. The work of Jenni Raitoharju was funded by the Academy of Finland (project 324475). Alexandros Iosifidis acknowledges funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 957337.

Rahoittajat	Rahoittajan numero
NSF-Business
Horizon 2020 Framework Programme	957337
Academy of Finland	324475

Julkaisufoorumi-taso

Jufo-taso 1

!!ASJC Scopus subject areas

Theoretical Computer Science
Yleinen tietojenkäsittelytiede

Pääsy asiakirjaan

10.1007/978-3-031-43415-0_13

Siteeraa tätä

Laakom, F., Raitoharju, J., Iosifidis, A., & Gabbouj, M. (2023). Learning Distinct Features Helps, Provably. teoksessa D. Koutra, C. Plant, M. Gomez Rodriguez, E. Baralis, & F. Bonchi (Toimittajat), Machine Learning and Knowledge Discovery in Databases: Research Track - European Conference, ECML PKDD 2023, Proceedings (Sivut 206-222). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vuosikerta 14170 LNAI). Springer. https://doi.org/10.1007/978-3-031-43415-0_13

Laakom, Firas ; Raitoharju, Jenni ; Iosifidis, Alexandros et al. / Learning Distinct Features Helps, Provably. Machine Learning and Knowledge Discovery in Databases: Research Track - European Conference, ECML PKDD 2023, Proceedings. Toimittaja / Danai Koutra ; Claudia Plant ; Manuel Gomez Rodriguez ; Elena Baralis ; Francesco Bonchi. Springer, 2023. Sivut 206-222 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{43d81a58d7f342cea0b974f853f91f7e,

title = "Learning Distinct Features Helps, Provably",

abstract = "We study the diversity of the features learned by a two-layer neural network trained with the least squares loss. We measure the diversity by the average -distance between the hidden-layer features and theoretically investigate how learning non-redundant distinct features affects the performance of the network. To do so, we derive novel generalization bounds depending on feature diversity based on Rademacher complexity for such networks. Our analysis proves that more distinct features at the network{\textquoteright}s units within the hidden layer lead to better generalization. We also show how to extend our results to deeper networks and different losses.",

keywords = "Feature Diversity, Generalization Theory, Neural Networks",

author = "Firas Laakom and Jenni Raitoharju and Alexandros Iosifidis and Moncef Gabbouj",

note = "Publisher Copyright: {\textcopyright} 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.; European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases ; Conference date: 18-09-2023 Through 22-09-2023",

year = "2023",

doi = "10.1007/978-3-031-43415-0_13",

language = "English",

isbn = "9783031434143",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer",

pages = "206--222",

editor = "Danai Koutra and Claudia Plant and {Gomez Rodriguez}, Manuel and Elena Baralis and Francesco Bonchi",

booktitle = "Machine Learning and Knowledge Discovery in Databases",

}

Laakom, F, Raitoharju, J, Iosifidis, A & Gabbouj, M 2023, Learning Distinct Features Helps, Provably. julkaisussa D Koutra, C Plant, M Gomez Rodriguez, E Baralis & F Bonchi (toim), Machine Learning and Knowledge Discovery in Databases: Research Track - European Conference, ECML PKDD 2023, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Vuosikerta 14170 LNAI, Springer, Sivut 206-222, European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, Turin, Italia, 18/09/23. https://doi.org/10.1007/978-3-031-43415-0_13

Learning Distinct Features Helps, Provably. / Laakom, Firas; Raitoharju, Jenni; Iosifidis, Alexandros et al.
Machine Learning and Knowledge Discovery in Databases: Research Track - European Conference, ECML PKDD 2023, Proceedings. toim. / Danai Koutra; Claudia Plant; Manuel Gomez Rodriguez; Elena Baralis; Francesco Bonchi. Springer, 2023. s. 206-222 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vuosikerta 14170 LNAI).

Tutkimustuotos: Konferenssiartikkeli › Tieteellinen › vertaisarvioitu

TY - GEN

T1 - Learning Distinct Features Helps, Provably

AU - Laakom, Firas

AU - Raitoharju, Jenni

AU - Iosifidis, Alexandros

AU - Gabbouj, Moncef

PY - 2023

Y1 - 2023

N2 - We study the diversity of the features learned by a two-layer neural network trained with the least squares loss. We measure the diversity by the average -distance between the hidden-layer features and theoretically investigate how learning non-redundant distinct features affects the performance of the network. To do so, we derive novel generalization bounds depending on feature diversity based on Rademacher complexity for such networks. Our analysis proves that more distinct features at the network’s units within the hidden layer lead to better generalization. We also show how to extend our results to deeper networks and different losses.

AB - We study the diversity of the features learned by a two-layer neural network trained with the least squares loss. We measure the diversity by the average -distance between the hidden-layer features and theoretically investigate how learning non-redundant distinct features affects the performance of the network. To do so, we derive novel generalization bounds depending on feature diversity based on Rademacher complexity for such networks. Our analysis proves that more distinct features at the network’s units within the hidden layer lead to better generalization. We also show how to extend our results to deeper networks and different losses.

KW - Feature Diversity

KW - Generalization Theory

KW - Neural Networks

U2 - 10.1007/978-3-031-43415-0_13

DO - 10.1007/978-3-031-43415-0_13

M3 - Conference contribution

AN - SCOPUS:85173212674

SN - 9783031434143

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 206

EP - 222

BT - Machine Learning and Knowledge Discovery in Databases

A2 - Koutra, Danai

A2 - Plant, Claudia

A2 - Gomez Rodriguez, Manuel

A2 - Baralis, Elena

A2 - Bonchi, Francesco

PB - Springer

T2 - European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases

Y2 - 18 September 2023 through 22 September 2023

ER -

Laakom F, Raitoharju J, Iosifidis A , Gabbouj M. Learning Distinct Features Helps, Provably. julkaisussa Koutra D, Plant C, Gomez Rodriguez M, Baralis E, Bonchi F, toimittajat, Machine Learning and Knowledge Discovery in Databases: Research Track - European Conference, ECML PKDD 2023, Proceedings. Springer. 2023. s. 206-222. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-43415-0_13

Learning Distinct Features Helps, Provably

Abstrakti

Julkaisusarja

Conference

Rahoitus

Julkaisufoorumi-taso

!!ASJC Scopus subject areas

Pääsy asiakirjaan

Sormenjälki

Siteeraa tätä