An implementation of a high throughput data ingestion system for machine logs in manufacturing industry

Jaehui Park, Su Young Chi

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

10 Scopus citations

Abstract

This paper aims at presenting a case study of designing and implementing a data ingestion system for manufacturers. In our implementation, clustered server architecture for high throughput data ingestion is proposed with regard to following factors: receiving stream data, i.e., machine logs, from a set of milling machines, storing them in a centralized messaging queue, and sinking to external systems with ease. Especially, we leverage the power of the open sources frameworks, Apache Kafka, Apache Hadoop File System and Apache Flume to cope with the data streams from a large number of machines in the factory floors. As this is an on-going study, we only illustrate our implementation details with structural diagrams, but exclude the theoretical study and the performance evaluation results in this paper.

Original languageEnglish
Title of host publicationICUFN 2016 - 8th International Conference on Ubiquitous and Future Networks
PublisherIEEE Computer Society
Pages117-120
Number of pages4
ISBN (Electronic)9781467399913
DOIs
StatePublished - 9 Aug 2016
Event8th International Conference on Ubiquitous and Future Networks, ICUFN 2016 - Vienna, Austria
Duration: 5 Jul 20168 Jul 2016

Publication series

NameInternational Conference on Ubiquitous and Future Networks, ICUFN
Volume2016-August
ISSN (Print)2165-8528
ISSN (Electronic)2165-8536

Conference

Conference8th International Conference on Ubiquitous and Future Networks, ICUFN 2016
Country/TerritoryAustria
CityVienna
Period5/07/168/07/16

Keywords

  • data ingestion system
  • machine logs
  • manufacturer
  • stream data

Fingerprint

Dive into the research topics of 'An implementation of a high throughput data ingestion system for machine logs in manufacturing industry'. Together they form a unique fingerprint.

Cite this