Cloud Platform Scalability & Optimisation
An intelligence-driven data science and behavior change company was experiencing scalability and performance bottlenecks in its existing data architecture. They needed to scale and optimize their platform to collect and analyze data from mobile devices.
The company had built an AWS-based cloud platform to process data from their mobile application SDK by collecting smartphone sensor data. This platform was designed according to a Lambda architecture with a speed layer for real-time processing and a batch layer for the daily processing of sensor data. While the platform leveraged cloud computing and modern technologies, there were still many performance and scalability bottlenecks, and this way, only a limited number of users were supported.
In only three months, Klarrio
Behind the scenes
Our customer built an AWS-based cloud platform to process data from their mobile application SDK by collecting smartphone sensor data.
It was designed according to a lambda architecture with both a speed layer for real-time processing and a batch layer for daily processing of sensor data.
After a cloud architecture analysis and load tests, we improved the messaging layer from 4k to 200k and supported persistent connections using Scala, Apache-Spark, Kafka and elasticsearch.
- Apache Spark
- Technical optimizations
- Data science code adjustments
- Messaging layer improvement
“At Klarrio we learned the hard way that the real scalability challenge goes beyond optimizing the CPU and memory of the current processing jobs. We found that the real issues lied in the resources that cannot be easily scaled up, such as network, single point of failures, recovering from failed components, read/write postgresql limitations, …
We really had to think outside the box here. Besides the technical optimizations, we also had to adjust the data science code written in Python, and solve queuing problems in the APIs between micro services as well as in Kafka caused by bursty traffic patterns.”
”Solving scalability issues without additional compute resources and costs is a tough, but fun challenge for any data engineer.Bruno De BusCTO
Want to work on similar projects?
Introverts and extroverts, geeks, nerds, and digital poets... Klarrio is the perfect place to learn and teach, experiment and brainstorm, exercise your brain, and feed your passion. Surrounded by people with amazing, world-changing talents.
We're your one-stop cloud-native partner
Big Data engineering. Data science. Data Analytics. Site Reliability Engineering. Consulting. And customized Open Source projects for companies of all sizes. What can Klarrio do for you today?
Just a few projects examples.