Neural networks are rapidly gaining popularity in chemical modeling and Quantitative Structure–Activity Relationship (QSAR) thanks to their ability to handle multitask problems. However, outcomes of neural networks depend on the tuning of several hyperparameters, whose small variations can often strongly affect their performance. Hence, optimization is a fundamental step in training neural networks although, in many cases, it can be very expensive from a computational point of view. In this study, we compared four of the most widely used approaches for tuning hyperparameters, namely, grid search, random search, tree-structured Parzen estimator, and genetic algorithms on three multitask QSAR datasets. We mainly focused on parsimonious optimization and thus not only on the performance of neural networks, but also the computational time that was taken into account. Furthermore, since the optimization approaches do not directly provide information about the influence of hyperparameters, we applied experimental design strategies to determine their effects on the neural network performance. We found that genetic algorithms, tree-structured Parzen estimator, and random search require on average 0.08% of the hours required by grid search; in addition, tree-structured Parzen estimator and genetic algorithms provide better results than random search.

Valsecchi, C., Consonni, V., Todeschini, R., Orlandi, M., Gosetti, F., Ballabio, D. (2021). Parsimonious optimization of multitask neural network hyperparameters. MOLECULES, 26(23) [10.3390/molecules26237254].

Parsimonious optimization of multitask neural network hyperparameters

Valsecchi, Cecile
Primo
;
Consonni, Viviana;Todeschini, Roberto;Orlandi, Marco Emilio;Gosetti, Fabio;Ballabio, Davide
2021

Abstract

Neural networks are rapidly gaining popularity in chemical modeling and Quantitative Structure–Activity Relationship (QSAR) thanks to their ability to handle multitask problems. However, outcomes of neural networks depend on the tuning of several hyperparameters, whose small variations can often strongly affect their performance. Hence, optimization is a fundamental step in training neural networks although, in many cases, it can be very expensive from a computational point of view. In this study, we compared four of the most widely used approaches for tuning hyperparameters, namely, grid search, random search, tree-structured Parzen estimator, and genetic algorithms on three multitask QSAR datasets. We mainly focused on parsimonious optimization and thus not only on the performance of neural networks, but also the computational time that was taken into account. Furthermore, since the optimization approaches do not directly provide information about the influence of hyperparameters, we applied experimental design strategies to determine their effects on the neural network performance. We found that genetic algorithms, tree-structured Parzen estimator, and random search require on average 0.08% of the hours required by grid search; in addition, tree-structured Parzen estimator and genetic algorithms provide better results than random search.
Articolo in rivista - Articolo scientifico
Genetic algorithms; Grid search; Neural networks; Optimization; Random search; Tree-structured Parzen estimator;
English
30-nov-2021
2021
26
23
7254
open
Valsecchi, C., Consonni, V., Todeschini, R., Orlandi, M., Gosetti, F., Ballabio, D. (2021). Parsimonious optimization of multitask neural network hyperparameters. MOLECULES, 26(23) [10.3390/molecules26237254].
File in questo prodotto:
File Dimensione Formato  
10281-337913_VoR.pdf

accesso aperto

Tipologia di allegato: Publisher’s Version (Version of Record, VoR)
Licenza: Creative Commons
Dimensione 2.86 MB
Formato Adobe PDF
2.86 MB Adobe PDF Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/10281/337913
Citazioni
  • Scopus 11
  • ???jsp.display-item.citation.isi??? 10
Social impact