Search Results

Assist Deploy Is Available on GitHub

Hello! For some time now I have been working on automating SSIS deployments, and earlier this week I published my efforts on GitHub . But before I get into the what/how, let’s focus on the why and let me catch you up on how I got here… The task to take an ispac and deploy in and of itself is quite a straightforward process as there are multiple ways to do this . For those of you who want the abridged version of the linked post, the choices are as follows: Integration Services Deploy Wizard SSIS Catalog T-SQL API PowerShell

Access Denied –How To Prevent a Failure Mid-Deploy

Hello! As part of any decent path-to-live, it is obviously crucial to deploy to other environments. This is crucial because not only do we need to test the changes being made, but just as importantly that the deployment will succeed. An ideally these environments should match as closely to production as possible. The less difference there is between production and all environments up to production, the greater the chances a deployment will succeed. Sounds simple, and perhaps even trivial, but if there is one thing that always, and I mean always catches deployments out, it’s permissions. Chances are production permissions

When To Use Octopus Deploy Script Modules

Hello! Lately I've been thinking a lot about script modules in Octopus Deploy. Script Modules, for those of you not ITK, are a collection of PowerShell cmdlets or functions that exist outside of a script step that can be used across multiple steps across different projects/environments. They don't have to be used to contain modules exclusively; they can just contain that will get executed upon the initial import of the module. The issue here is that these script modules are imported or every step in a process. So unless the code you write is idempotent, you'll probably cause an error

SSDT And Pre-Model Scripts Part 2: Execute Pre-Model Scripts Using Octopus Deploy

Hello! Part one of this post concerned the notion of SSDT and Pre-Model Scripts . This post relates to an actual implementation of executing a SQL script using Octopus Deploy. My pull request on Github was approved, so my step template can be exported from the Octopus Library . The only part that I feel requires some explanation is the SQL Scripts and the option to select a previous step. If you've read part one of this post you'll know that you keep the pre-model scripts in a folder in the database solution. These are not compiled (ie not in

ARM Template Snippet - Allow Azure Resources To Access Azure SQL Instance With This One Simple Trick!

  Hello! I’m really getting into using ARM Templates to deploy Azure Resources. For those of you not ITK, and for the sake of brevity I’ll summarise in a sentence, ARM Templates are basically Infrastructure as Code. It’s all very interesting, go and have a read . In addition to having our code in source, the ability for us to deploy our entire infrastructure from source into Azure is fantastic, because now we can run one single script to build deploy the infrastructure and the code from source, and we can all have our own personal environment in Azure. And

Team City Meta Runner: Get Build Number

One of the neat things about TeamCity is that it gives you a great deal of control over your process. TeamCity is great to use as a control flow for your pipeline, especially if you are using Octopus to deploy. But one of the pain points here is keeping the parity between build/deploy versions in Octopus the same as in TeamCity. It's important because it prevents users from having to jump between the UI's to keep track of what is deployed where, especially if you are automating the whole process. There's several ways to keep the build numbers aligned, the

SSDT and Pre-Model Scripts

Hello! pre-deploy scripts in SSDT scripts may be executed before the main diff script is executed, but not before the diff script is generated. This is an important fact to understand . If you want scripts to be executed before the diff is generated, then you need to execute pre-model scripts. How you go about this is up to you, but there needs to be a few rules that must be followed: Idempotent : big word, and is used in maths. But don't let that put you off. What it means in this context is that a script can be

AssistDeploy Samples Repo Released

Hello! So AssistDeploy has been out for a little while now and so I decided to release the project I was using for testing during development. At the moment it uses just one scenario I used for testing as opposed to the several that I ran through. This is because initially I just want to provide a straightforward working sample. What became quite a big challenge was the fact that you need more than just an Integration Services project. So I've included the database projects which will allow the dtsx package to run successfully post-deploy. AssistDeploySamples The set-up process takes

Azure and Guest OS Families

Recently one of our clients came to us with an issue: the Azure SDK 2.7.8 was being retired and needed updating for their Azure Cloud Services. OK, simple enough, but the issue was that the latest versions of the Azure SDK require .NET 4.6.2 installed to work. You see, when you deploy a Cloud Service to Azure, it deploys it to a VM that is spun up as part of the deployment process. And by default, Windows Server 2012 R2 does not come with .NET 4.6.2 installed. So the problem was, how do you get 4.6.2 installed on that flavour

SQL Supper Scripts

Hello! Thanks to everyone who turned up yesterday at SQL Supper: there was a good turnout of both new and familiar faces. The Demo Gods were with me and I was able to log on to both my Azure VM and able to deploy to SQL Azure. I’ve uploaded the scripts to gist and shared below. I also spoke about raising a Connect Issue so that Microsoft.Build.Utilities.Core NuGet package will work with Microsoft.Data.Tools.Msbuild. I’d like to see this so that we do not have to install the Microsoft Build Tools 2015 MSI on the box. And this is important because

SQL Agent depLoyment Tasks Out Now!

Hello! In my time since leaving university, I’ve worked across the spectrum, from tester to DBA, it has always been abundantly clear that SQL Agent Jobs are one of those things that are really difficult to get into source control and deploy. Sure, you can script them out, but that doesn’t really factor in changes. And in many places, the phrase “whatever is in prod” has often been the answer to the question “what are the SQL Agent Jobs supposed to look like?” And frequently, relying on msdb being backed up as been the backup process for SQL Agent Jobs.

Migrating SSIS Packages to SSIS Azure part Two – Automating the Deployment

Hello! If you’ve read and followed through my previous post, you will have World Wide Importers Integration Services project running in SSIS Azure. It’s all very interesting, go and have a read . One thing that is missing form that guide, the documentation, and SSIS in general, is how to automate SSIS Deployments. In the WWI SSIS project, there are connection managers that we had to manually update the values of to get it to work post-deploy. This is exactly the opposite of what we want to do. Back when SQL Server 2012 was known as Denali, one of the

Fix For Using Azure Active Directory and DacFX

Hello! As part of an SSDT project we have a contained user that authenticates against an Azure Active Directory group (read more on the CREATE USER page). However the account we are executing deployments with is the SQL Admin account on the Azure SQL Instance. And so we get this error - The executed script: CREATE USER [myUser] FOR EXTERNAL PROVIDER; ' Reason: '' At line:94 char:13 + Throw $toThrow + ~~~~~~~~~~~~~~ + CategoryInfo : OperationStopped: (Deployment fail... ' Reason: '':String) [], RuntimeException + FullyQualifiedErrorId : Deployment failed: 'Could not deploy package. Error SQL72014: .Net SqlClient Data Provider: Msg 33159,

SSDTPokedex: Integrating Slack and VSTS Into GitHub Repo - An Infinite Improvement

Hello! One of the home projects I’m currently working on is migrating a database over from SQLite to SQL Server. There’s several tasks that need to be accomplished before we can say that this is successful. Broadly speaking they fit into the key pillars of successful software development: Plan Develop Deploy Measure   So the development part is well under way: there is a repo in GitHub of an SSDT solution that will compile locally (it works on my machine anyway.) Plus I have a couple of releases: one intentionally broken and one fixed. Now there’s plenty of tasks I

How To Use MSBuild Arguments in Visual Studio Builds

When creating MSBuild.proj files for builds in Visual Studio, there are times when you might want some targets called, and other times you don't want the targets called. In my case, our Production builds compile all the web and windows service solutions as well asthe deployment scripts for databases and our BIDS projects. Essentially everything I would want if I was going to run an entire deployment for a release. However when deploying a dll hotfix, I don't want the SQL scripts and BIDS stuff built. When we want to deploy a hotfix, speed is of the essence. Our build

Home

Plan Develop Deploy Measure Sabin.io is a Data Engineering practice. Our focus is on helping companies deliver data systems. We help companies build sustainable applications that consider the support and ongoing development. The use of agile methodologies is fantastic but needs to be aligned with the engineering practices of testing, continuous integration/delivery and a feedback loop to enable continual improvement. Our experience stretches from windows server management and automation through to the support and management of BI systems. We have strong application development experience which means we are able to deliver and support solutions that work with all those involved

How To Compile SQLProj Files Using Cmdline MSBuild... Errors Included!

I recently needed to build and deploy about 40 small database projects that were in 4 or 5 different database solutions. And I needed to do this several times a day, so compiling via Visual Studio would be a boring and tedious process. So to speed up the process I decided to write the build process in an MSBuild target file and call initiate the build process through PowerShell. The targets file was simple enough to put together. This would be saved in the root location of all the solution folders as "BuildAllDBProjects.targets.xml". Then the PowerShell would be simple enough;

SSIS Package Execution In Azure Is Now Available

Well, it’s been some time coming but SSIS packages are the latest product to make the move from on premise to Azure. You can now take your SSIS projects and deploy them to the new Platform as a Service (PaaS) offering in Azure. The aim of the team at Microsoft was for users to take their current SSIS packages and just “lift and shift” these to Azure. So in development terms that means that there are minimum to no changes to be made in the solution at least. But before we get into the deployment and running of SSIS packages

Keeping The Database Dev Ops Overhead Lightweight

Hello! One very important aspect of Dev Ops that is perhaps over-looked is the overhead that comes with adopting Dev Ops practices. To help explain what I mean, let’s break that sentence down a bit. What Do I Mean By “Dev Ops Practices” I have a strong suspicion that for each of the posts for this T-SQL Tuesday on Database Dev Ops, everyone will have a slightly different take. Or rather, they are going to articulate what Dev Ops means to them. And so here is my take: broadly speaking, Dev Ops is about increasing the cadence of a feature

Deploying SSDT Scripts (With My Good Eye Closed)

(If anyone actually gets the reference: kudos. If not, then it's just that you're not a Soundgarden fan) So, there's something that has been troubling me about deploying to databases with SSDT for some time. Actually, it's not a SSDT-exclusive problem per se, it's more of a database/release management process. What it pertains to is that most people like to create the deployment script for review prior to deploying to production. This makes sense if you are new to the Continuous Deployment process, and that maybe sometimes SSDT creates scripts that are not quite expected. You may skip this because

Join Us This Wednesday at The Impact Hub King's Cross

…Where I will be presenting on the headless SSDT release from Microsoft. So up front, this does not sound like a topic which requires much discussion: There’s a NuGet package. But it is what underlies the usage of this NuGet package is where the discussion is. When it comes to build servers, the idea behind what software that needs to be installed to support builds is generally “whatever is on the developers box, that also needs to be installed here.” And that is the way I used ot think. However, this idea upfront seems simple, but if you are supporting

AssistDeploy 1.2 Is Now Live

Evening! AssistDeploy , our attempt to fully automate SSIS Deployments, is not yet a week old, yet we’re already on release 1.2 , which is our 4th release. If you’re wondering how we can be on a 4th release, when the number is .2, I suggest you have a read up on SemVer . Unlike the previous post , this will be brief. But like that post, I’m going to delve into why I’ve made the changes before what, so that the context is understood. What most IT projects attempt to achieve is take some knowledge of a subject matter

T SQL Tuesday: Shipping Database Changes with SSDT

Hello! Let’s see how this goes: this months subject for T SQL Tuesday is about shipping database changes, something we here are all familiar with. So I thought I’d make some notes about a tool I’m very familiar with, SQL Server Data Tools. The Good It’s free! SSDT works with Visual Studio Community up to Ultimate, and from Visual Studio 2015 onwards it comes with it’s own Visual Studio IDE. SSDT Has a NuGet package available. So you don’t need to install Visual Studio to get builds running, and crucially can control which version is used to compile at a

Continuous Integration with Jenkins, SQL Server and Windows Containers

Why use Windows Containers? When creating database applications we need consistency in all our environments to ensure quality releases. Traditionally developers might have their own instance of SQL Server on their workstation to develop against. Database projects would be created in SSDT and pushed to source control when ready for testing. If you’re not using SSDT for database development already, then you should seriously consider it to make your life easier and increase the quality of your releases. Ed Elliot explains why in this blog post . A problem with CI for databases is that databases are a shared resource

Uploading Files To Data Lake Storage With PowerShell Part Two

Carrying on from our previous post on automating the process of uploading files to  Azure Data Lake Store , we will check if a Resource Group exists, and if it does not then it will create it. Although the entire script is available on Git (posted below) I’m going to go into one function per post so that I can go in greater depth. Part One of this blog series focused on logging in to an Azure Subscription. Today’s function starts on line 42 and is called Set-AzureResourceGroup. Before we go into it though, I want to take a moment

SSDT 16.5 Released

Hello! Recently the SQL Tools Team released a new version of both SQL Server Data Tools ( SSDT ) and SQL Server Management Studio (SSMS.)  There’s a range of bug fixes, but two new features that I am particualrly interested in. Firstly, SQLPAckage.exe and the DacFx API can now generate deployment report, deployment script and publish to a datbase all in one action. Neat! This is useful because it’s important to keep track of exactly what has changed on a database. Of course there’s nothing stopping you right now from creating executin these options in three separate actions, but there

SSDT 16.5 Released part 2: Using the DacFx API and Samples!

Hello! Yesterday I posted about the new release of SSDT from the SQL Tools Team at Microsoft. Two of the big changes are the ability to create the deployment report, deployment script and execute the deployment all in one command. The other change is that now for Azure two scripts are generated: one for any changes that need a connection to master, and the other script for changes to the user database. The samples yesterday showed how to execute the new method using SQLPackage, but a lot of people, myself included, have automated the deployment using the DacFx API through

Why is SSDT Always Rebuilding My Constraints?

Hello! Let me begin by saying that I’m a big fan of SSDT. It’s free, it works with all flavours of Visual Studio, the team do their very best to keep up-to-date with features that are released with increasing regularity by Microsfot in both Azure and SQl Server on-prem. I’ve met the team a few times, and they’re genuinely keen to engage with the users of SSDT in how it can be better, and how it can be extensible. So, SSDT is a great tool. I said great, but not perfect. It has it's limitations, the same as any tool.

SSDTPokedex: Migrating a Database Into SSDT

Hello!   If you want to have the best chance of something being successful, you have to be committed to it from the start. That’s a pretty fatuous sounding statement, almost as bad as “to make something better you have to do more of the good stuff and less of the bad stuff”, so let me contextualise: if you desire to have good testing coverage on an application, then you need to be serious about testing form the first day you write code for the application. Be it manual or automated testing, you need to put the effort in terms

Azure Powershell 4.0 may break your scripts

Ensuring backwards compatibility is something that one has to consider very carefully when doing continuous delivery. We are all to well aware of the challenges of this with database systems as, generally, the database lives much longer than the apps that interact with it and thus one has to maintain the data. SQL has far too many “legacy features” that can’t be changed due to potential breaking changes. Thankfully the SQL team now have a more robust way of managing change and that’s through the compatibility level for the database. This allows you to upgrade to the latest runtime but

Migrating SSIS Packages to SSIS Azure

Hello! In case you missed the announcement (and there were a lot of announcements during MSIgnite), SQL Server Integration Services is in Public Preview on Azure! I’ve written about it elsewhere in greater depth , but here are the headlines: It makes use of SSIS Scale Out , which was released as part of SQL Server 2017 . Although it is based on SSIS Scale Out, you can’t actually configure SSIS Scale Out to run on the instance. If this confuses you then read my in-depth post. SSISDB is installed in either SQL Azure or on a Managed Instance. You

Automating SQL Server Performance Testing

You run performance tests as well as functional tests when deploying new code changes to SQL Server, right? Not many people do, I think you should, and this article will show you how to do it by harnessing an existing performance tool, rather than writing your own monitoring infrastructure from scratch. Any good performance monitoring tool that records information to a database will do fine, and we prefer to use Sentry One . Here are the steps to accomplish this. Create a baseline database When you release your database change, you want to have something to compare against as an

Git: How to work on a feature branch but pull in later commits from another branch?

On a client site last week the question was raised: I want to work on a feature for a project which will take longer than other people merging their branches into the dev branch. I therefore want to merge any changes on the dev branch into mine, test my changes before pushing back to the dev branch, and then and ultimately master for a release. We are using a git flow methodology whereby the development manager will merge feature branches from developers’ branches via pull requests into the dev branch. Tests are run and if they pass, we merge into