Continuous delivery of data services has always been a battle for us data analytics peeps. Things are getting better when we think about business intelligence and dev ops, however there are still gaps. This blog post is an account of one such issue I recently faced and how I worked around the problem.
Problem
While boot strapping an analytics solution with Azure Logic Apps we discovered the ARM template output for the service from the Azure portal was less than adequate. The Azure Logic Apps user interface is really friendly in the portal and allows you to develop with ease. Unfortunately though, when you export the JSON in the browser you’ll find key value pairs and arrays created for almost everything you clicked on the design canvas, regardless of use when the service is deployed. Plus, you’ll see a whole load of things you never even used/clicked and connection strings to almost everything in your subscription! Maybe helpful in the designer, trying to pre-empt what you’ll need. But not helpful when using the template and moving between development, test and production.
Firstly, you can avoid the verbose ARM template by using the Logic Apps plugin for Visual Studio. Get the extension from Visual Studio > Tools > Extensions and Updates. It is not installed by default as part of the Azure SDK.
Once installed, use the Cloud Explorer panel to open the logic app in the local designer (really cool having a local designer). Download the content of the app to give you an ARM template than reflects what triggers and connections you actually used in the service.
Workaround
Break out the PowerShell!
Next, depending on the complexity of your logic app and the connection strings you have there are likely to be 3 main things you’ll need to alter in the ARM template before you move into test or production.
- Perform a blanket string replacement on resource group names and services throughout the JSON. For example ‘DevLogicApp01’ to ‘ProdLogicApp01’. I accept a find and replace is a little dirty but its the easiest way to simultaneously switch your keys and values between environments throughout the template. Parsing the JSON at this point and doing targeted replacements would take much more effort so I think a string replace is the best thing for it. This of course also assumes you have strong naming conventions for your resources.
- Inject a new set of parameters at the top of the ARM template to be edited and used when moving between environments.
- Replace key value content from static values to reference the new parameters created in point 2.
Let’s look at the PowerShell I used to perform each of these steps.
One – String Replacements
For this I simply treated the JSON as a single string and used the PowerShell -replace switch combine with an array of all the values I needed to change. In the looping anything added to the variable $Values will be iterated over and replace called for the old and new parameters.
#Read in your JSON
$RawTemplateJSON = Get-Content -Path "C:\Temp\RawARMTemplate.json"
$NewTemplateJSON = $RawTemplateJSON #| ConvertFrom-Json
#Create an array for your replacements
$Values = @()
#Values to replace:
[Pscustomobject]$Value1 = @{
Old=”DevLogicApp01″
New=”ProdLogicApp01″
}
$Values += $Value1
[Pscustomobject]$Value2 = @{
Old=”DevResourceGroup01″
New=”ProdResourceGroup01″
}
$Values += $Value2
#Loop over values to replace:
ForEach($Value in $Values)
{
$NewTemplateJSON = $NewTemplateJSON -replace $Value.Old, $Value.New
}
#Output new JSON template
$Output = $NewTemplateJSON
$Output | Out-File “C:\Temp\NewARMTemplate.json”
Two – New Local Parameters in the ARM Template
Similar to the method in number one I’ve created a values variable and iterate over the parameters. This time though I parse the JSON and use the ‘Add-Member’ condition to add the JSON snippet against the required parent.
#Read JSON file
$RawTemplateJSON = Get-Content -Path "C:\Temp\RawARMTemplate.json"
#Parse raw JSON
$NewTemplateJSON = $RawTemplateJSON | ConvertFrom-Json
$Parameters = @()
#New parameters
[Pscustomobject]$Parameter1 = @{
Name=”app_Name”
Type=”string”
DefaultValue=””
Description=”Logic app name”
}
$Parameters += $Parameter1
[Pscustomobject]$Parameter2 = @{
Name=”connectionString1″
Type=”string”
DefaultValue=””
Description=”Connection string for first connection”
}
$Parameters += $Parameter2
#Display table (for debugging)
#$Parameters.ForEach({[PSCustomObject]$_}) | Format-Table -AutoSize
#Inject new parameters into ARM template
ForEach ($Parameter in $Parameters)
{
$Type = $Parameter.Type
$DefaultValue = $Parameter.DefaultValue
$Description = $Parameter.Description
$NewParam =@”
{
“type”:”$Type”,
“defaultValue”: “$DefaultValue”,
“metadata”: {
“description”: “$Description”
}
}
“@
#Add member to JSON
$NewTemplateJSON.parameters | Add-Member -Name $Parameter.Name -value (Convertfrom-Json $NewParam) -MemberType NoteProperty
} #end foreach
#Deal with escaped characters
$Output = $NewTemplateJSON | ConvertTo-Json -Depth 50 | % { [System.Text.RegularExpressions.Regex]::Unescape($_) }
#Output new JSON template
$Output | Out-File “C:\Temp\NewARMTemplate.json”
Three – Replace Key Values
Lastly, perhaps the most difficult content to handle, the static values against various keys. These will be tricky because depending on what your logic app is doing will depend on the JSON structure presented. In the example below I simply deal with the logic app name. This is found within the ‘resources’ parent. However, if you have a value within an array. Or, a value at the top level of the structure the PowerShell required will differ. Check out Stack Overflow for additional support if your unsure of the syntax. This question is a good start:
#Read JSON file
$RawTemplateJSON = Get-Content -Path "C:\Temp\RawARMTemplate.json"
#Value to replace
$AppName = “LogicApp01”
$ParamAppName = “app_Name”
#Parse raw JSON
$NewTemplateJSON = $RawTemplateJSON | ConvertFrom-Json
#Replace static value with param call
$NewTemplateJSON.resources | Where{$_.name -eq $AppName} | ForEach{$_.name = “[parameters(‘” + $PAppName + “‘)]”}
#Deal with escaped characters
$Output = $NewTemplateJSON | ConvertTo-Json -Depth 50 | % { [System.Text.RegularExpressions.Regex]::Unescape($_) }
#Output new template
$Output | Out-File “C:\Temp\NewARMTemplate.json”
Conclusion
Azure ARM templates are excellent for handling the deployment of services between environments. However, in most cases I’ve found they aren’t dynamic enough for the complexity of real world services.
The above approach using PowerShell to assist can help and of course be applied to other services. This isn’t confined to Azure Logic Apps.
If PowerShell isn’t your thing, maybe try the ARM native functions to inject into your template. Link: https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-template-functions
Many thanks for reading
Welcome to the Blog & Website of Paul Andrew 