Incremental PARAFAC decomposition for three-dimensional tensors using apache spark

Hye Kyung Yang; Hwan Seung Yong

doi:10.1007/978-3-030-19274-7_5

Incremental PARAFAC decomposition for three-dimensional tensors using apache spark

Hye Kyung Yang, Hwan Seung Yong

Computer Science & Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

3 Scopus citations

Abstract

Recent studies have focused on the use of tensor analysis for tensor decomposition because this method can identify more latent factor and patterns, compared to the matrix factorization approach. The existing tensor decomposition studies used static dataset in their analyses. However, in practice, data change and increase over time. Therefore, this paper proposes an incremental Parallel Factor Analysis (PARAFAC) tensor decomposition algorithm for three-dimensional tensors. The method of incremental tensor decomposition can reduce recalculation costs associated with the addition of new tensors. The proposed method is called InParTen; it performs distributed incremental PARAFAC tensor decomposition based on the Apache Spark framework. The proposed method decomposes only new tensors and then combines them with existing results without recalculating the complete tensors. In this study, it was assumed that the tensors grow with time as the majority of the dataset is added over a period. In this paper, the performance of InParTen was evaluated by comparing the obtained results for execution time and relative errors against existing tensor decomposition tools. Consequently, it has been observed that the method can reduce the recalculation cost of tensor decomposition.

Original language	English
Title of host publication	Web Engineering - 19th International Conference, ICWE 2019, Proceedings
Editors	In-Young Ko, Maxim Bakaev, Flavius Frasincar
Publisher	Springer Verlag
Pages	63-71
Number of pages	9
ISBN (Print)	9783030192730
DOIs	https://doi.org/10.1007/978-3-030-19274-7_5
State	Published - 2019
Event	19th International Conference on Web Engineering, ICWE 2019 - Daejeon, Korea, Republic of Duration: 11 Jun 2019 → 14 Jun 2019

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	11496 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	19th International Conference on Web Engineering, ICWE 2019
Country/Territory	Korea, Republic of
City	Daejeon
Period	11/06/19 → 14/06/19

Bibliographical note

Funding Information:
Acknowledgement. This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF-2016R1D1A1B03931529)

Funding Information:
This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF-2016R1D1A1B03931529).

Publisher Copyright:
© Springer Nature Switzerland AG 2019.

Keywords

Apache spark
Incremental tensor decomposition
PARAFAC decomposition

Access to Document

10.1007/978-3-030-19274-7_5

Cite this

Yang, H. K., & Yong, H. S. (2019). Incremental PARAFAC decomposition for three-dimensional tensors using apache spark. In I-Y. Ko, M. Bakaev, & F. Frasincar (Eds.), Web Engineering - 19th International Conference, ICWE 2019, Proceedings (pp. 63-71). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11496 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-030-19274-7_5

Yang, Hye Kyung ; Yong, Hwan Seung. / Incremental PARAFAC decomposition for three-dimensional tensors using apache spark. Web Engineering - 19th International Conference, ICWE 2019, Proceedings. editor / In-Young Ko ; Maxim Bakaev ; Flavius Frasincar. Springer Verlag, 2019. pp. 63-71 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{d60e9d67115c44c188fe6fcefd03b283,

title = "Incremental PARAFAC decomposition for three-dimensional tensors using apache spark",

abstract = "Recent studies have focused on the use of tensor analysis for tensor decomposition because this method can identify more latent factor and patterns, compared to the matrix factorization approach. The existing tensor decomposition studies used static dataset in their analyses. However, in practice, data change and increase over time. Therefore, this paper proposes an incremental Parallel Factor Analysis (PARAFAC) tensor decomposition algorithm for three-dimensional tensors. The method of incremental tensor decomposition can reduce recalculation costs associated with the addition of new tensors. The proposed method is called InParTen; it performs distributed incremental PARAFAC tensor decomposition based on the Apache Spark framework. The proposed method decomposes only new tensors and then combines them with existing results without recalculating the complete tensors. In this study, it was assumed that the tensors grow with time as the majority of the dataset is added over a period. In this paper, the performance of InParTen was evaluated by comparing the obtained results for execution time and relative errors against existing tensor decomposition tools. Consequently, it has been observed that the method can reduce the recalculation cost of tensor decomposition.",

keywords = "Apache spark, Incremental tensor decomposition, PARAFAC decomposition",

author = "Yang, {Hye Kyung} and Yong, {Hwan Seung}",

note = "Funding Information: Acknowledgement. This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF-2016R1D1A1B03931529) Funding Information: This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF-2016R1D1A1B03931529). Publisher Copyright: {\textcopyright} Springer Nature Switzerland AG 2019.; 19th International Conference on Web Engineering, ICWE 2019 ; Conference date: 11-06-2019 Through 14-06-2019",

year = "2019",

doi = "10.1007/978-3-030-19274-7_5",

language = "English",

isbn = "9783030192730",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Verlag",

pages = "63--71",

editor = "In-Young Ko and Maxim Bakaev and Flavius Frasincar",

booktitle = "Web Engineering - 19th International Conference, ICWE 2019, Proceedings",

}

Yang, HK & Yong, HS 2019, Incremental PARAFAC decomposition for three-dimensional tensors using apache spark. in I-Y Ko, M Bakaev & F Frasincar (eds), Web Engineering - 19th International Conference, ICWE 2019, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11496 LNCS, Springer Verlag, pp. 63-71, 19th International Conference on Web Engineering, ICWE 2019, Daejeon, Korea, Republic of, 11/06/19. https://doi.org/10.1007/978-3-030-19274-7_5

Incremental PARAFAC decomposition for three-dimensional tensors using apache spark. / Yang, Hye Kyung; Yong, Hwan Seung.
Web Engineering - 19th International Conference, ICWE 2019, Proceedings. ed. / In-Young Ko; Maxim Bakaev; Flavius Frasincar. Springer Verlag, 2019. p. 63-71 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11496 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Incremental PARAFAC decomposition for three-dimensional tensors using apache spark

AU - Yang, Hye Kyung

AU - Yong, Hwan Seung

N1 - Funding Information: Acknowledgement. This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF-2016R1D1A1B03931529) Funding Information: This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF-2016R1D1A1B03931529). Publisher Copyright: © Springer Nature Switzerland AG 2019.

PY - 2019

Y1 - 2019

N2 - Recent studies have focused on the use of tensor analysis for tensor decomposition because this method can identify more latent factor and patterns, compared to the matrix factorization approach. The existing tensor decomposition studies used static dataset in their analyses. However, in practice, data change and increase over time. Therefore, this paper proposes an incremental Parallel Factor Analysis (PARAFAC) tensor decomposition algorithm for three-dimensional tensors. The method of incremental tensor decomposition can reduce recalculation costs associated with the addition of new tensors. The proposed method is called InParTen; it performs distributed incremental PARAFAC tensor decomposition based on the Apache Spark framework. The proposed method decomposes only new tensors and then combines them with existing results without recalculating the complete tensors. In this study, it was assumed that the tensors grow with time as the majority of the dataset is added over a period. In this paper, the performance of InParTen was evaluated by comparing the obtained results for execution time and relative errors against existing tensor decomposition tools. Consequently, it has been observed that the method can reduce the recalculation cost of tensor decomposition.

AB - Recent studies have focused on the use of tensor analysis for tensor decomposition because this method can identify more latent factor and patterns, compared to the matrix factorization approach. The existing tensor decomposition studies used static dataset in their analyses. However, in practice, data change and increase over time. Therefore, this paper proposes an incremental Parallel Factor Analysis (PARAFAC) tensor decomposition algorithm for three-dimensional tensors. The method of incremental tensor decomposition can reduce recalculation costs associated with the addition of new tensors. The proposed method is called InParTen; it performs distributed incremental PARAFAC tensor decomposition based on the Apache Spark framework. The proposed method decomposes only new tensors and then combines them with existing results without recalculating the complete tensors. In this study, it was assumed that the tensors grow with time as the majority of the dataset is added over a period. In this paper, the performance of InParTen was evaluated by comparing the obtained results for execution time and relative errors against existing tensor decomposition tools. Consequently, it has been observed that the method can reduce the recalculation cost of tensor decomposition.

KW - Apache spark

KW - Incremental tensor decomposition

KW - PARAFAC decomposition

UR - http://www.scopus.com/inward/record.url?scp=85065461778&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-19274-7_5

DO - 10.1007/978-3-030-19274-7_5

M3 - Conference contribution

AN - SCOPUS:85065461778

SN - 9783030192730

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 63

EP - 71

BT - Web Engineering - 19th International Conference, ICWE 2019, Proceedings

A2 - Ko, In-Young

A2 - Bakaev, Maxim

A2 - Frasincar, Flavius

PB - Springer Verlag

T2 - 19th International Conference on Web Engineering, ICWE 2019

Y2 - 11 June 2019 through 14 June 2019

ER -

Yang HK, Yong HS. Incremental PARAFAC decomposition for three-dimensional tensors using apache spark. In Ko I-Y, Bakaev M, Frasincar F, editors, Web Engineering - 19th International Conference, ICWE 2019, Proceedings. Springer Verlag. 2019. p. 63-71. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-19274-7_5

Incremental PARAFAC decomposition for three-dimensional tensors using apache spark

Abstract

Publication series

Conference

Bibliographical note

Keywords

Access to Document

Fingerprint

Cite this