Image for post
Image for post

If you are using Azure Data Factory (ADF) and need to access data outside of Azure you need to install an Integration Runtime (IR) on a Virtual Machine (VM) that does have access to the data. At ASOS we have some sources that are still on premises or within VNETs in which case you need an IR. We are always evolving our implementations and recently had to split one of our solutions into two, which meant we had to reconfigure the IRs in multiple environments to connect to different ADFs in other Azure subscriptions.

The current recommendation of re-configuring an instance is that you cannot enter another key and you will need to reinstall it. However, there is a much easier way to do this using a single line of PowerShell. …

Azure Portal Corrupts Downloaded Files

Image for post
Image for post

Have you ever had a situation where you’re working with a tester who has reported a bug, and you can’t for the life of you recreate it? It does after all work on your own machine…

We had an example of when downloading a parquet file using the Azure Portal Data Explorer, it corrupted the file and made it unreadable by Spark and parq, however downloading the same file with the Azure Storage Explorer solved the problem.

We were two days from deploying to production and our tester raised a bug, reporting he could not read the file he downloaded using the Azure Portal Data Explorer. …

Image for post
Image for post

In the Advanced Analytics team here at ASOS, we have a few infrequently executed (monthly or weekly) Python processes. They were built by technical users, not engineers, and therefore used to run on a physical machine somewhere in a cupboard. For obvious reasons they had to be moved to a more stable and manageable infrastructure.

We had a requirement to run these Python scripts as part of an ADF (Azure Data Factory) pipeline and react on completion of the script. …


Eugene Niemand

Lead Data QA Engineer at - I have a passion for Test Driven Development, Agile Methodologies, Continuous Integration and Delivery using Microsoft Azure

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store