[TBD]CNN approaches for data visualisation

ELK

About

• • Supports JSON input data format

• Logstash acts as an aggregator — pulling data from various sources before pushing it down the pipeline, usually into Elasticsearch but also into a buffering component in larger production environments.

input { file { type => "json" path => "/logs/mylogs.log" codec => "json" } } output { file { path => "/logs/out.log" } }

Components

Helper tools

• Beat can be used as helper for log stash. For example, for adjusting back pressure

• • Real time Messaging system

    • Latency is < 10 ms

  • Use-cases

    • Netflix - Recommendations in real-time

    • Uber - To gather user, tax detail in real time

    • Linkedin -

      • Spam filter,

      • User interaction for better connection recommendation in real time

  • Preferred input data format is AVRO. Refer here

Below is the flow with ELK

Zipkin

• • Provides visualisation for distributed tracing

OpenTracing

• OpenTracing is an open-source standard API for consistent distributed tracing of requests across

  • processes,

  • from web and

  • mobile client platforms to the storage systems and custom backends at the bottom of an application stack.

  • Jaeger is open source implementation from Uber

Splunk

Reference

https://images.app.goo.gl/fFwp3xekAyMwPPoi6

https://stackoverflow.com/questions/22941739/using-json-with-logstash

https://zipkin.io

https://images.app.goo.gl/APbHkdfap6buo7MS9

https://logz.io/blog/filebeat-vs-logstash/

https://images.app.goo.gl/q5Fs7rfSQc6ej1sb6

https://www.elastic.co/beats/

https://images.app.goo.gl/pjptajRv3Sxp6aGM9

https://images.app.goo.gl/FLPSnMQVq4MmrSKM7

https://opentracing.io/docs/best-practices/instrumenting-frameworks/