It is a common pattern that you design an entire ETL/ELT solution inside Databricks notebooks and use Azure Data Factory to orchestrate an entire process. Today we will investigate how to pass an ADF parameter/variable into a Databricks notebook. A link to a code repository is placed at the end…

When you think about logs in an Azure context you say Log Analytics Workspace. There is another option baked in Azure Synapse, Synapse Data Explorer. A link to a code repository is placed at the end of the article. The first thing you need to do is creating a Data…

If you want to speed up your tables refresh process you can partition a table. Then update only partitions where data has been changed. Power BI offers more convenient way to do this process — Hybrid tables. After you imported a table. Set two parameters a [RangeStart] and a [RangeEnd]…

Since Microsoft introduced a Datamart we can say that Power BI ecosystem is a full cloud experience. Power BI Desktop in the cloud I can state. How it works? Once you create and open a Datamart you are welcomed by well-known ribbon where you can work with data.

A while ago I showed how to hide/insert a measure through a slicer. Here is the post: Power BI — show and hide measures on slicer and visual Microsoft has published a baked in functionality Field Parameter, so we don’t need to use workarounds and entire process is much simplified…

In that short post I descripted Power BI deployment pipelines. Power BI — deployment pipelines Today I show you how to manage a dataset connection using parameters and Deployment Rules. There are 6 workspaces: 3 for reports and 3 for datasets and they are split into 3 stages: DEV, INT…

This post is a previous one continuation: Databricks MLflow — manage models Without further ado, here are a few examples of using MLflow Python API. An easy way to remove a run or an experiment.

Let’s assume that you have you data prepared and you want to use MLflow to manage and use you ML model. First step is to set an experiment.

Today, I would like to do a short intro about Databricks SQL Warehouses (formerly SQL endpoint). What is it? Simply, a place where you can work with hive tables through SQL. I can highlight three main functionalities: work with data through SQL, create dashboards/visualizations and alerts based on your data. …