Back to overview

Data integration with Microsoft Fabric: Insight and practical applications

Technical contribution

July 12, 2024

Microsoft Fabric provides a comprehensive data platform that covers various areas such as data integration, data engineering, data science, data warehousing, real-time data analysis and business intelligence. This platform enables the collection, processing, storage and analysis of data in an integrated environment. In this article, we will give you an insight into the topic of data integration using Microsoft Fabric. You will receive practical application examples that illustrate how Microsoft Fabric can improve your data processing processes.

Key components of Fabric: ADF, Power BI, Synapse & Data Activator

The basis of the Software-as-a-Service (SaaS) platform in the Azure Cloud is the OneLake, a lake-based architecture that acts as a central data storage, similar to OneDrive. By connecting different storage locations to a single lake, data can be used without time-consuming moving or copying between systems. OneLake is based on Azure Data Lake Storage (ADLS) and enables data to be stored in various file formats. The Delta Parquet format is used for table data. Microsoft Fabric offers a comprehensive platform for data analysis through the integration of various Azure services. These include, among others:

Azure Data Factory for data integration,
Power BI for data analysis and
Azure Synapse for data warehousing, data transformations with Spark, data science with Azure Machine Learning and real-time analysis of large amounts of data.

Data integration with Lakehouse as scalable data storage

Microsoft Fabric is based on a lakehouse that is based on the scalable storage layer of OneLake and uses Apache Spark and SQL compute engines for big data processing. A lakehouse combines elements of data warehouses and data lakes and offers flexible data storage for files and tables that can be queried using SQL. Lakehouses combine the SQL-based analysis functions of a relational data warehouse with the flexibility and scalability of a data lake. Companies can store structured and unstructured data in a central repository and use it for analysis purposes. Delta Lake-formatted tables support ACID transactions to ensure the integrity of the data.

Data management in Microsoft Fabric

Thanks to central storage in OneLake, governanceand security policies can be easily created and controlled via the Admin Center. This includes managing user groups and permissions, configuring data sources and gateways, and monitoring usage and performance. In addition, confidentiality designations from Microsoft Purview Information Protection are used to classify and protect confidential data.

ETL process in Fabric

Microsoft Fabric offers various options for data integration. From simple uploads of small data sets to the development of professional ETL routes with Azure Data Factory and Dataflows Gen2 within the Fabric environment. By automating ETL processes, data from different sources can be connected, transformed and loaded into a lakehouse.

Data analysis in Microsoft Fabric

The Lakehouse architecture makes it possible to query integrated data from various formats using SQL. For complex analyses, notebooks can be created in Fabric that support programming in PySpark, Spark SQL, Spark R and Scala. Spark can be used to efficiently process large data sets.

Power BI reports can be easily connected to the data in OneLake, either directly in Fabric or used as a desktop application. Direct Lake enables the loading of files in Parquet format directly from a data lake for analyzing large amounts of data in Power BI.

Licenses and costs of Microsoft Fabric

Microsoft Fabric offers capacity and single user licenses. F or P capacity licenses and at least one single user license are required to collaborate and share content. Capacity licenses are divided into Stock Keeping Units (SKUs) and offer different fabric resources. Single user licenses determine the available features and include free, pro and premium single user licenses.

Comprehensive platform for data integration and more

Microsoft Fabric is a comprehensive platform for data integration, data engineering, data science, data warehousing, real-time analysis and business intelligence. The diverse components and functions of this platform offer numerous options for collecting, processing, storing and analyzing data.