An implementation of a high throughput data ingestion system for machine logs in manufacturing industry

Jaehui Park; Su Young Chi

doi:10.1109/ICUFN.2016.7536997

An implementation of a high throughput data ingestion system for machine logs in manufacturing industry

Jaehui Park, Su Young Chi

Department of Statistics

Electronics and Telecommunications Research Institute

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

11 Scopus citations

Abstract

This paper aims at presenting a case study of designing and implementing a data ingestion system for manufacturers. In our implementation, clustered server architecture for high throughput data ingestion is proposed with regard to following factors: receiving stream data, i.e., machine logs, from a set of milling machines, storing them in a centralized messaging queue, and sinking to external systems with ease. Especially, we leverage the power of the open sources frameworks, Apache Kafka, Apache Hadoop File System and Apache Flume to cope with the data streams from a large number of machines in the factory floors. As this is an on-going study, we only illustrate our implementation details with structural diagrams, but exclude the theoretical study and the performance evaluation results in this paper.

Original language	English
Title of host publication	ICUFN 2016 - 8th International Conference on Ubiquitous and Future Networks
Publisher	IEEE Computer Society
Pages	117-120
Number of pages	4
ISBN (Electronic)	9781467399913
DOIs	https://doi.org/10.1109/ICUFN.2016.7536997
State	Published - 9 Aug 2016
Event	8th International Conference on Ubiquitous and Future Networks, ICUFN 2016 - Vienna, Austria Duration: 5 Jul 2016 → 8 Jul 2016

Publication series

Name	International Conference on Ubiquitous and Future Networks, ICUFN
Volume	2016-August
ISSN (Print)	2165-8528
ISSN (Electronic)	2165-8536

Conference

Conference	8th International Conference on Ubiquitous and Future Networks, ICUFN 2016
Country/Territory	Austria
City	Vienna
Period	5/07/16 → 8/07/16

Keywords

data ingestion system
machine logs
manufacturer
stream data

Access to Document

10.1109/ICUFN.2016.7536997

Cite this

Park, J., & Chi, S. Y. (2016). An implementation of a high throughput data ingestion system for machine logs in manufacturing industry. In ICUFN 2016 - 8th International Conference on Ubiquitous and Future Networks (pp. 117-120). Article 7536997 (International Conference on Ubiquitous and Future Networks, ICUFN; Vol. 2016-August). IEEE Computer Society. https://doi.org/10.1109/ICUFN.2016.7536997

@inproceedings{b8897d80f0694c89a7edb3075fed7c6f,

title = "An implementation of a high throughput data ingestion system for machine logs in manufacturing industry",

abstract = "This paper aims at presenting a case study of designing and implementing a data ingestion system for manufacturers. In our implementation, clustered server architecture for high throughput data ingestion is proposed with regard to following factors: receiving stream data, i.e., machine logs, from a set of milling machines, storing them in a centralized messaging queue, and sinking to external systems with ease. Especially, we leverage the power of the open sources frameworks, Apache Kafka, Apache Hadoop File System and Apache Flume to cope with the data streams from a large number of machines in the factory floors. As this is an on-going study, we only illustrate our implementation details with structural diagrams, but exclude the theoretical study and the performance evaluation results in this paper.",

keywords = "data ingestion system, machine logs, manufacturer, stream data",

author = "Jaehui Park and Chi, {Su Young}",

note = "Publisher Copyright: {\textcopyright} 2016 IEEE.; 8th International Conference on Ubiquitous and Future Networks, ICUFN 2016 ; Conference date: 05-07-2016 Through 08-07-2016",

year = "2016",

month = aug,

day = "9",

doi = "10.1109/ICUFN.2016.7536997",

language = "English",

series = "International Conference on Ubiquitous and Future Networks, ICUFN",

publisher = "IEEE Computer Society",

pages = "117--120",

booktitle = "ICUFN 2016 - 8th International Conference on Ubiquitous and Future Networks",

address = "United States",

}

Park, J & Chi, SY 2016, An implementation of a high throughput data ingestion system for machine logs in manufacturing industry. in ICUFN 2016 - 8th International Conference on Ubiquitous and Future Networks., 7536997, International Conference on Ubiquitous and Future Networks, ICUFN, vol. 2016-August, IEEE Computer Society, pp. 117-120, 8th International Conference on Ubiquitous and Future Networks, ICUFN 2016, Vienna, Austria, 5/07/16. https://doi.org/10.1109/ICUFN.2016.7536997

An implementation of a high throughput data ingestion system for machine logs in manufacturing industry. / Park, Jaehui; Chi, Su Young.
ICUFN 2016 - 8th International Conference on Ubiquitous and Future Networks. IEEE Computer Society, 2016. p. 117-120 7536997 (International Conference on Ubiquitous and Future Networks, ICUFN; Vol. 2016-August).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - An implementation of a high throughput data ingestion system for machine logs in manufacturing industry

AU - Park, Jaehui

AU - Chi, Su Young

PY - 2016/8/9

Y1 - 2016/8/9

N2 - This paper aims at presenting a case study of designing and implementing a data ingestion system for manufacturers. In our implementation, clustered server architecture for high throughput data ingestion is proposed with regard to following factors: receiving stream data, i.e., machine logs, from a set of milling machines, storing them in a centralized messaging queue, and sinking to external systems with ease. Especially, we leverage the power of the open sources frameworks, Apache Kafka, Apache Hadoop File System and Apache Flume to cope with the data streams from a large number of machines in the factory floors. As this is an on-going study, we only illustrate our implementation details with structural diagrams, but exclude the theoretical study and the performance evaluation results in this paper.

AB - This paper aims at presenting a case study of designing and implementing a data ingestion system for manufacturers. In our implementation, clustered server architecture for high throughput data ingestion is proposed with regard to following factors: receiving stream data, i.e., machine logs, from a set of milling machines, storing them in a centralized messaging queue, and sinking to external systems with ease. Especially, we leverage the power of the open sources frameworks, Apache Kafka, Apache Hadoop File System and Apache Flume to cope with the data streams from a large number of machines in the factory floors. As this is an on-going study, we only illustrate our implementation details with structural diagrams, but exclude the theoretical study and the performance evaluation results in this paper.

KW - data ingestion system

KW - machine logs

KW - manufacturer

KW - stream data

UR - http://www.scopus.com/inward/record.url?scp=84983316702&partnerID=8YFLogxK

U2 - 10.1109/ICUFN.2016.7536997

DO - 10.1109/ICUFN.2016.7536997

M3 - Conference contribution

AN - SCOPUS:84983316702

T3 - International Conference on Ubiquitous and Future Networks, ICUFN

SP - 117

EP - 120

BT - ICUFN 2016 - 8th International Conference on Ubiquitous and Future Networks

PB - IEEE Computer Society

T2 - 8th International Conference on Ubiquitous and Future Networks, ICUFN 2016

Y2 - 5 July 2016 through 8 July 2016

ER -

An implementation of a high throughput data ingestion system for machine logs in manufacturing industry

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this