[MS] Part 5 – Unlock the Power of Azure Data Factory: A Guide to Boosting Your Data Ingestion Process - devamazonaws.blogspot.com

John Folberth and Joe Fitzgerald share sample guidance for developing and deploying an Azure Data Factory into multiple environments.

---

Introduction 

Part 1

1. Architecture and Scenario
2. Creating resources in Azure
3. Create Azure Storage Containers
4. Create Azure Key Vaults
5. Create Azure Data Factory: With Key Vault Access

Part 2 

1. Configure Azure Data Factory Source Control
2. Construct Azure Data Factory Data Pipeline
3. Publishing Concept for Azure Data Factory
4. Configure Deployed Azure Resources.

Part 3

1. The YAML Pipeline Structure
2. The Publish Process
3. ARM Template Parameterization
4. ADF ARM Template Deployment

Part 4

1. How to use Azure DevOps Pipeline Templates

This section will build on that and include how to build/deploy larger Data Factories which leverage Azure Resource Manager Linked Templates for deployment.   

When Linked Templates Are Required 

Typically, users will not desire to use linked templates; however, there are explicit limitations of a single ARM template which may require one to deploy their ARM template via Linked Templates. Specifically, when the ARM template.json file is > 4MB is one such instance.   Regardless of the Data Factory size the Data Factory build process will generate the necessary files and file structure to accommodate both a single template deployment and a linked ARM template deployment. The linked templates are only required if your main ARM template file is greater than >4MB. (i.e. if you have a larger Data Factory).   

What is a Linked Template 

A linked template is a process by which the main deployment template will call additional files (‘linked’). These files will need to stored in an Azure Storage Account. This will break up the single 4 MB file limitation of an ARM template as it will call multiple files whose size is < 4MB. Now to be clear you will still be limited by limitation of an ARM template. As a reminder those are: 

• 256 parameters 
• 256 variables 
• 800 resources (including copy count) 
• 64 output values 

• 10 unique locations per subscription/tenant/management group scope 
• 24,576 characters in a template expression 

Check out full post (Part 5) and the series in the Healthcare and Life Sciences Tech Community here.
Post Updated on October 31, 2023 at 07:13AM
Thanks for reading
from devamazonaws.blogspot.com