{"id":1394,"date":"2025-09-07T17:35:42","date_gmt":"2025-09-07T17:35:42","guid":{"rendered":"https:\/\/dataforma.tech\/?p=1394"},"modified":"2025-09-07T17:35:44","modified_gmt":"2025-09-07T17:35:44","slug":"otimizando-o-gerenciamento-de-memoria-no-pyspark-guia-pratico-para-notebooks-databricks","status":"publish","type":"post","link":"https:\/\/dataforma.tech\/en\/blog\/otimizando-o-gerenciamento-de-memoria-no-pyspark-guia-pratico-para-notebooks-databricks\/","title":{"rendered":"Optimizing Memory Management in PySpark: A Practical Guide for Databricks Notebooks"},"content":{"rendered":"
\"\"<\/figure>\n\n\n\n

Anyone who's worked with PySpark knows how frustrating it is when an application crashes due to lack of memory. You're there, processing data peacefully, when suddenly everything stops and that error message no one wants to see appears. It turns out that memory management in PySpark can reduce resource consumption by up to 2x when you know what you're doing.<\/p>\n\n\n\n

Until recently, figuring out memory issues in distributed PySpark applications was like looking for a needle in a haystack, especially when the problem was in the Spark executors. You were left guessing which part of the code was consuming all the available memory.<\/p>\n\n\n\n

Fortunately, since Databricks Runtime 12.0, this headache has been greatly reduced with the arrival of memory analysis tools directly on the executors.<\/p>\n\n\n\n

Today, we have access to memory profiling tools that show exactly which lines of code in your UDFs are hogging resources. This completely changes how we can make the necessary improvements.<\/p>\n\n\n\n

There is also Adaptive Query Execution, known as AQE, which came with Spark 3.x and helps improve query execution plans while they are running, using the statistics it collects.<\/p>\n\n\n\n

The question remains: how can we use these tools in a practical way to improve the performance of our notebooks in Databricks?<\/p>\n\n\n\n

We'll see how to better organize your data layout, how to manage shuffles without breaking everything, how to solve those annoying skew issues, and how to use cache in a way that really makes a difference in your application's performance.<\/p>\n\n\n\n

Organize Data and Write Better in Delta Tables<\/h3>\n\n\n\n
\"\"
Image Source:\u00a0BigData Boutique<\/a><\/figcaption><\/figure>\n\n\n\n

You've probably experienced that situation where a query that should be fast ends up taking forever. Often the culprit is "small files" in Delta Lake. When you're writing data frequently, especially in streams or batch updates, thousands of small files end up creating a mess in the metadata and slowing down your queries.<\/p>\n\n\n\n

Fortunately, Databricks has some very useful features to solve this data layout issue in Delta tables.<\/p>\n\n\n\n

delta.autoOptimize.optimizeWrite<\/a> and delta.autoOptimize.autoCompact<\/h3>\n\n\n\n

There is a feature called optimizeWrite<\/strong> which reduces the number of files written, making each file larger during write operations. The idea is simple: it takes several small writes to the same partition and combines them into a single operation before executing, creating larger, more efficient files.<\/p>\n\n\n\n

To enable this functionality, you have two options:<\/p>\n\n\n\n