The clientâs offering, a data observability platform, allows teams to discover, prioritise, and collaboratively resolve data issues on both batch- and streaming platforms through a single pane of glass.
Challenge
The client is specialized in state-of-the-art data quality monitoring in batch processing environments. Since they identified a number of their clients transitioning to streaming architectures they needed a solution to provide a comprehensive solution capable of addressing all of their clientsâ needs.
Solution
The number of use cases which use some form of real-time data processing has increased drastically over the last decades. Data is generated at an unprecedented volume, velocity and variety. This brings up the concern of guaranteeing data quality of these generated streams. To provide a solution for this, an automated streaming data quality monitoring system was created. This system generates real-time insights on the quality of data and visualizes it in a single pane of glass.
The Technology behind
- Flink
- Kafka
- Scala
”âThings work out best for those who make the best of how things work out.â
â John Wooden
Results
Data Streaming | An initial Alpha version was created that allowed the client to offer an open source data quality analysis on Streaming data sources Â
Dashboard | Single pane of glass for both batch and streaming use cases.
Ask for a demo!
We’re your one-stop cloud-native partner
Big Data engineering. Data science. Data Analytics. Site Reliability Engineering. Consulting. And customized Open Source projects for companies of all sizes. Learn more about what Klarrio can do for you today.
Contact us
Other Projects
Just a few projects examples.
