Azure Data Engineering Cookbook. Design and implement batch and streaming analytics using Azure Cloud Services

Ahmad Osama

Data engineering is one of the faster growing job areas as Data Engineers are the ones who ensure that the data is extracted, provisioned and the data is of the highest quality for data analysis. This book uses various Azure services to implement and maintain infrastructure to extract data from multiple sources, and then transform and load it for data analysis.It takes you through different techniques for performing big data engineering using Microsoft Azure Data services. It begins by showing you how Azure Blob storage can be used for storing large amounts of unstructured data and how to use it for orchestrating a data workflow. You'll then work with different Cosmos DB APIs and Azure SQL Database. Moving on, you'll discover how to provision an Azure Synapse database and find out how to ingest and analyze data in Azure Synapse. As you advance, you'll cover the design and implementation of batch processing solutions using Azure Data Factory, and understand how to manage, maintain, and secure Azure Data Factory pipelines. You’ll also design and implement batch processing solutions using Azure Databricks and then manage and secure Azure Databricks clusters and jobs. In the concluding chapters, you'll learn how to process streaming data using Azure Stream Analytics and Data Explorer.By the end of this Azure book, you'll have gained the knowledge you need to be able to orchestrate batch and real-time ETL workflows in Microsoft Azure.

Azure Data Engineering Cookbook. Get well versed in various data engineering techniques in Azure using this recipe-based guide - Second Edition

Nagaraj Venkatesan, Ahmad Osama

The famous quote 'Data is the new oil' seems more true every day as the key to most organizations' long-term success lies in extracting insights from raw data. One of the major challenges organizations face in leveraging value out of data is building performant data engineering pipelines for data visualization, ingestion, storage, and processing. This second edition of the immensely successful book by Ahmad Osama brings to you several recent enhancements in Azure data engineering and shares approximately 80 useful recipes covering common scenarios in building data engineering pipelines in Microsoft Azure.You’ll explore recipes from Azure Synapse Analytics workspaces Gen 2 and get to grips with Synapse Spark pools, SQL Serverless pools, Synapse integration pipelines, and Synapse data flows. You’ll also understand Synapse SQL Pool optimization techniques in this second edition. Besides Synapse enhancements, you’ll discover helpful tips on managing Azure SQL Database and learn about security, high availability, and performance monitoring. Finally, the book takes you through overall data engineering pipeline management, focusing on monitoring using Log Analytics and tracking data lineage using Azure Purview.By the end of this book, you’ll be able to build superior data engineering pipelines along with having an invaluable go-to guide.

Azure Data Factory Cookbook. Build and manage ETL and ELT pipelines with Microsoft Azure's serverless data integration service

Dmitry Anoshin, Dmitry Foshin, Roman Storchak, Xenia Ireton

Azure Data Factory (ADF) is a modern data integration tool available on Microsoft Azure. This Azure Data Factory Cookbook helps you get up and running by showing you how to create and execute your first job in ADF. You’ll learn how to branch and chain activities, create custom activities, and schedule pipelines. This book will help you to discover the benefits of cloud data warehousing, Azure Synapse Analytics, and Azure Data Lake Gen2 Storage, which are frequently used for big data analytics. With practical recipes, you’ll learn how to actively engage with analytical tools from Azure Data Services and leverage your on-premise infrastructure with cloud-native tools to get relevant business insights. As you advance, you’ll be able to integrate the most commonly used Azure Services into ADF and understand how Azure services can be useful in designing ETL pipelines. The book will take you through the common errors that you may encounter while working with ADF and show you how to use the Azure portal to monitor pipelines. You’ll also understand error messages and resolve problems in connectors and data flows with the debugging capabilities of ADF.By the end of this book, you’ll be able to use ADF as the main ETL and orchestration tool for your data warehouse or data platform projects.

Azure Data Factory Cookbook. Build ETL, Hybrid ETL, and ELT pipelines using ADF, Synapse Analytics, Fabric and Databricks - Second Edition

Dmitry Foshin, Tonya Chernyshova, Dmitry Anoshin, Xenia Ireton

This new edition of the Azure Data Factory book, fully updated to reflect ADS V2, will help you get up and running by showing you how to create and execute your first job in ADF. There are updated and new recipes throughout the book based on developments happening in Azure Synapse, Deployment with Azure DevOps, and Azure Purview. The current edition also runs you through Fabric Data Factory, Data Explorer, and some industry-grade best practices with specific chapters on each.You’ll learn how to branch and chain activities, create custom activities, and schedule pipelines, as well as discover the benefits of cloud data warehousing, Azure Synapse Analytics, and Azure Data Lake Gen2 Storage. With practical recipes, you’ll learn how to actively engage with analytical tools from Azure Data Services and leverage your on-premises infrastructure with cloud-native tools to get relevant business insights. You'll familiarize yourself with the common errors that you may encounter while working with ADF and find out the solutions to them. You’ll also understand error messages and resolve problems in connectors and data flows with the debugging capabilities of ADF.By the end of this book, you’ll be able to use ADF with its latest advancements as the main ETL and orchestration tool for your data warehouse projects.

Azure Data Scientist Associate Certification Guide. A hands-on guide to machine learning in Azure and passing the Microsoft Certified DP-100 exam

Andreas Botsikas, Michael Hlobil

The Azure Data Scientist Associate Certification Guide helps you acquire practical knowledge for machine learning experimentation on Azure. It covers everything you need to pass the DP-100 exam and become a certified Azure Data Scientist Associate.Starting with an introduction to data science, you'll learn the terminology that will be used throughout the book and then move on to the Azure Machine Learning (Azure ML) workspace. You'll discover the studio interface and manage various components, such as data stores and compute clusters.Next, the book focuses on no-code and low-code experimentation, and shows you how to use the Automated ML wizard to locate and deploy optimal models for your dataset. You'll also learn how to run end-to-end data science experiments using the designer provided in Azure ML Studio.You'll then explore the Azure ML Software Development Kit (SDK) for Python and advance to creating experiments and publishing models using code. The book also guides you in optimizing your model's hyperparameters using Hyperdrive before demonstrating how to use responsible AI tools to interpret and debug your models. Once you have a trained model, you'll learn to operationalize it for batch or real-time inferences and monitor it in production.By the end of this Azure certification study guide, you'll have gained the knowledge and the practical skills required to pass the DP-100 exam.

Azure Databricks Cookbook. Accelerate and scale real-time analytics solutions using the Apache Spark-based analytics service

Phani Raj, Vinod Jaiswal

Azure Databricks is a unified collaborative platform for performing scalable analytics in an interactive environment. The Azure Databricks Cookbook provides recipes to get hands-on with the analytics process, including ingesting data from various batch and streaming sources and building a modern data warehouse.The book starts by teaching you how to create an Azure Databricks instance within the Azure portal, Azure CLI, and ARM templates. You’ll work through clusters in Databricks and explore recipes for ingesting data from sources, including files, databases, and streaming sources such as Apache Kafka and EventHub. The book will help you explore all the features supported by Azure Databricks for building powerful end-to-end data pipelines. You'll also find out how to build a modern data warehouse by using Delta tables and Azure Synapse Analytics. Later, you’ll learn how to write ad hoc queries and extract meaningful insights from the data lake by creating visualizations and dashboards with Databricks SQL. Finally, you'll deploy and productionize a data pipeline as well as deploy notebooks and Azure Databricks service using continuous integration and continuous delivery (CI/CD).By the end of this Azure book, you'll be able to use Azure Databricks to streamline different processes involved in building data-driven apps.

Azure DevOps Explained. Get started with Azure DevOps and develop your DevOps practices

Sjoukje Zaal, Stefano Demiliani, Amit Malik

Developing applications for the cloud involves changing development methodologies and procedures. Continuous integration and continuous deployment (CI/CD) processes are a must today, but are often difficult to implement and adopt. Azure DevOps is a Microsoft Azure cloud service that enhances your application development life cycle and enables DevOps capabilities.Starting with a comprehensive product overview, this book helps you to understand Azure DevOps and apply DevOps techniques to your development projects. You'll find out how to adopt DevOps techniques for your development processes by using built-in Azure DevOps tools. Throughout the course of this book, you'll also discover how to manage a project with the help of project management techniques such as Agile and Scrum, and then progress toward development aspects such as source code management, build pipelines, code testing and artifacts, release pipelines, and GitHub integration. As you learn how to implement DevOps practices, this book will also provide you with real-world examples and scenarios of DevOps adoption.By the end of this DevOps book, you will have learned how to adopt and implement Azure DevOps features in your real-world development processes.

Azure DevOps Server 2019 Cookbook. Proven recipes to accelerate your DevOps journey with Azure DevOps Server 2019 (formerly TFS) - Second Edition

Tarun Arora, Utkarsh Shigihalli

Previously known as Team Foundation Server (TFS), Azure DevOps Server is a comprehensive on-premise DevOps toolset with a rich ecosystem of open source plugins. This book will help you learn how to effectively use the different Azure DevOps services.You will start by building high-quality scalable software targeting .NET, .NET Core and Node.js applications. Next, you will learn techniques that will help you to set up end-to-end traceability of your code changes, from design through to release. Whether you are deploying software on-premise or in the cloud in App Service, Functions, or Azure VMs, this book will help you learn release management techniques to reduce failures. As you progress, you will be able to secure application configuration by using Azure Key Vault. You will also understand how to create and release extensions to the Azure DevOps marketplace and reach the million-strong developer ecosystem for feedback. Later, the working extension samples will even allow you to iterate changes in your extensions easily and release updates to the marketplace quickly.By the end of this book, you will be equipped with the skills you need to break down the invisible silos between your software development teams, and transform them into a modern cross-functional software development team.