Category: Data Engineering

If you want your Python script to run daily, it might seem as simple as setting a time and starting it. However, it’s not that straightforward as most Python environments lack built-in scheduling features. There’s a range of advice out there, with common suggestions often involving complex cloud services, which are overkill for simple tasks.…

Data Engineers often face the challenge of Jupyter Notebooks crashing when loading large datasets into Pandas DataFrames. This problem signals a need to explore alternatives to Pandas for data processing. While common solutions like processing data in chunks or using Apache Spark exist, they come with their own complexities. In this post, we’ll examine these…

When managing data pipelines, there’s this crucial step that can’t be overlooked: defining a PySpark schema upfront. It’s a safeguard to ensure every new batch of data lands consistently. But if you’ve ever wrestled with creating Spark schemas manually, especially for those intricate JSON datasets, you know that it’s challenging and time-consuming. In this post,…

Business Intelligence (BI) Implementations go wrong more often than right. I’ve experienced this first hand and this post is going to outline the top challenges that get in the way of a successfully deployed dashboard at a lean tech startup.  In this post, BI encompasses reports and dashboards used for internal and external (customer-facing) purposes. 

Aggregating data from multiple sources into a centralized place can be a challenging task when creating reports. In the early stages, many software engineering teams tend to rely on familiar tools, often their application databases. Since the majority of data for tech startups is generated from their apps, it may seem logical to incorporate additional…

Traditional BI approaches have primarily centered around manual report generation, focusing on historical numerical data. This often leaves business teams longing for insights and grappling with the complexities of unstructured text data. However, AI-powered tools are poised to reshape how businesses gather, analyze, and interpret data. In this blog post, I will dive into four…

As an application scales, data volumes and complexity grow, necessitating the need for scalable data infrastructure. Faced with this challenge, the decision between building a custom solution or purchasing a ready-made service is more than just a technical choice; it’s a strategic dilemma that significantly affects operational agility, cost efficiency, and long-term scalability. In this…