Synapse deployment pipelines made simple(r)

Edo Terzić

/ 2024-05-26

Introduction

In the rapidly evolving world of IT, deployment pipelines are the backbone of seamless development and release processes. However, many teams currently wrestle with deployment systems that are anything but user-friendly.

For Synapse deployments these systems are cumbersome, fraught with bugs, and come with documentation that does more to confuse than clarify and is often either incomplete or straight up wrong. Additionally, it comes with either limited customization options, or makes customization an overly complex ordeal that can bog down even the most straightforward projects.

Recognizing these challenges, our team pursued a better solution. We needed a deployment pipeline that was not only reliable but also adaptable to our specific requirements without the added complexity. Our efforts led us to choose a more streamlined approach to Synapse deployment pipelines which is simple, works, and offers unlimited customization.

Architecture

center-big

The high-level architecture above seeks to emphasize a couple of important concepts we needed to consider during the design, alongside some caveats. Firstly, after connecting the Synapse workspace to GIT, all the changes inside the Synapse workspace are propagated to the repository. The important part here, as outlined by the bidirectional arrow, is that the reverse is true as well. Any changes made to the contents of the files themselves are, after publishing, propagated to the Synapse workspace as well.

With the disclaimer that in our use case we needed to have some differences between environments, and they needed to be up together at the same time so merging was not an option, this leaves us with the best approach of simply modifying the files themselves during the transition in the development lifecycle while marking the files that needed to stay with a certain naming convention (_DEV suffix for example).

Setup

The setup itself is not complicated and the key components are:

  • Connecting Synapse to the Azure repos.

center-big

  • Creating an Azure DevOps pipeline to migrate and modify the contents of the Synapse config files between environments

center-big

  • Examining the json config files and deciding on the edits you want to make during transition

center-big

Code

For the example code we will take the simplest of the config files, the blob trigger, so it is easy to understand. The rest of these files are modified in the exact same way, but of course with different parameters. Here is a thoroughly commented code (the variables have of course been replaced with placeholders)

center-big

center-big

Conclusion

The code is being executed in Azure DevOps simply because Azure Repos are easy to integrate with Synapse and we can use the system managed identity for easy authentication. You can absolutely replicate this approach anywhere using GitHub as a repository if that is where your code is stored. In this example, after much frustration with the azure API, we can see that the simplest solution is often the best as was the case for our project. This method allows us to modify the config files, and manipulate the deployments between environments without any limitations, and all that is achieved with code that is very simple to both visualize and execute.

Want to dive deeper into DevOps best practices and insights? Check out our blog for more valuable content!

Share This Story, Choose Your Platform!

Share This Story

Drive your business forward!

iOLAP experts are here to assist you