Deployment of Batch Processing for Log File Analysis

Tutkimustuotos: Conference contributionScientificvertaisarvioitu


We have used log file analysis in mining expected
behavior in intelligent transportation systems involving spatial
and temporal data. The challenge is how to extract complex
behavior from multiple traces, in which linear log analysis
proceeding in a row by row order does not suffice. Complex
Event Processing (CEP) is close to our need, but it is surprisingly
difficult to set up and deploy general purpose frameworks to the
purpose. This paper originates from the need to compare our
custom LOGDIG tool to Apache Flink. This paper focuses on the
deployment effort of the two, for which reason we consider
setting up the development and run-time environments, selecting
the proper analysis approach and evaluating the difficulty in five
different aspects. While LOGDIG is written solely in Python,
Flink is a combination of many languages, libraries, packages
and tools. Our comparison includes Flink in batch and stream
processing modes using external and internal preprocessing. We
lend the Degree of Difficulty (DoD) measure from sports to assess
the deployment effort. Flink needs significant setup effort for
deploying the same functionality as LOGDIG. The former is
continuously developing while LOGDIG is more focused and
stable and can be used more easily off-the-self.
Otsikko3RD IEEE International Conference on Industrial Cyber-Physical Systems
ISBN (painettu)978-1-7281-6389-5
DOI - pysyväislinkit
TilaJulkaistu - 12 kesäkuuta 2020
OKM-julkaisutyyppiA4 Artikkeli konferenssijulkaisussa
TapahtumaIEEE International Conference on Industrial Cyber-Physical Systems - Tampere, Suomi
Kesto: 10 kesäkuuta 202012 kesäkuuta 2020


ConferenceIEEE International Conference on Industrial Cyber-Physical Systems


  • Jufo-taso 1


Sukella tutkimusaiheisiin 'Deployment of Batch Processing for Log File Analysis'. Ne muodostavat yhdessä ainutlaatuisen sormenjäljen.

Siteeraa tätä