Microsoft Azure Services — overview and notes

A live page with raw notes about Azure services that I happen to use, read about or am just interested in.

This page will grow and change over time. It’s not meant to be “documentation”, but rather a quick list of important notes and high-level concepts that I wrote down for myself, in pretty much random order. Feel free to suggest additions and point out mistakes in the comments below or directly highlighting the incriminated content.

Image taken from DotNetTricks.com

Azure Services

• App Service Web Apps for N Tier monolithic apps. Web apps are just like web servers (IIS) on the cloud. See the first 20 mins of this video for a demo of an Angular front-end + ASP.NET MVC API being lifted-and-shifted to App Service…the rest of the video is about Azure Functions instead.
• Service Fabric and AKS (Azure Kubernetes Service) is more for microservice-based architectures.
• Azure Mobile Apps allow to quickly create database-drive backend services for mobile apps, and generate client-side libraries with support for offline sync. They do push notifications and other stuff.
• Azure Functions, aka serverless computing (don’t worry about installing and patching the OS, handle outages and replication, scale infrastructure, etc.). Small pieces of code triggered in a number of ways (http call, new file being written in blob storage, new message in Service Bus etc.). Provides auto-scaling. Azure Functions Core Tools allows you to develop, run and debug these locally from Visual Studio…but an online IDE is also available.
  Azure Functions Proxies allow you to forward requests to other external endpoints (also modifying the request/response parameters and headers), therefore being able to break a large API into multiple function apps (as in a microservice architecture), while still presenting a single API surface for clients.
  Example of hybrid app (website on App Service + MVC API on App Service + API endpoint on Azure Functions + Azure Functions proxy) from min 20 of this video.
  Read everything by Jeff Hollan (Senior PM for Azure Functions) to know much more about all this.
• Logic apps for backend jobs. Orchestrate API calls and tasks together with a visual designer. Can stitch together a number of Azure Functions together. It’s like running a workflow triggered by something (100+ connectors available).
• Azure Virtual Machines: used for “lift-and-shift” approach of current apps run on premise. It does have some auto-scaling capability, but you have to patch the OS yourself.

Containers

Service Fabric supports .NET Framework apps, Core apps, Linux and Windows Docker containers.

• Install Service Fabric Tools in Visual Studio to have specific templates for new projects. When running the projects, it actually deploys it to a local node of Service Fabric running on the dev machine. Navigate to https://localhost:19080/Explorer to load the browser-based Service Fabric Explorer and manage your local dev cluster (replace localhost with the remote IP address if the cluster is running somewhere else, and secure it).
• A Service Fabric app is mostly a regular .NET/Core app, with optional classes to intercept service-events (eg: service starting up or shutting down).
• While Azure is the easiest option to host and run Service Fabric clusters (during dev, just hit Publish from VS itself), you could also deploy to a on-premise infrastructure, or to Amazon AWS / EC2 instances. Read here.
• Quick-start app

AKS supports Core apps and Linux-based Docker containers. Win-based Docker container will be added. .NET Framework apps are not supported.

Even Azure AI models can be exported to containers that can run in AKS.

Announced at Build 2018 (and currently in private beta), Dev Spaces allows to deploy and *debug* AKS-hosted containers from Visual Studio.

Data Storage

Azure SQL DB, PostgreSQL and MySql for relational data.

• Azure SQL is like SQL Server but on the cloud. It’s marketed as “Put your database on autopilot”. Can be used with Server Explorer, SQL Server Management Studio etc. Full backups are automatically and by default done every 24h and incremental backups happen every 5m. Can be accessed by Entity Framework (Core or regular version). Offers automatic performance analysis (eg: suggestions to add an idex, use parametrised queries etc.), error reporting (eg: statements that reference invalid columns) and thread detection (eg: potential sql injection). Data Migration Assistant tool available to move data and object from on-premise to cloud.

Cosmos DB for unstructured data + small and large data

• Cosmos DB is a new version of DocumentDB. Geographical replication in realtime over different regions: 99.99% SLA for one region (data is replicated 4 times per data center) or 99.999% SLA for 2+ regions (30–40 regions available in total). Automatic indexing, limitless scalability (only pay for the throughput and storage you need), traffic management, multi-master capability for unlimited write scalability, multiple conflict handling options (eg: last write wins). Best option of globally distributed DBs, just select regions to replicate data to from a map of the world. Can be used from .NET Framework and Core with the Microsoft.Azure.DocumentDB(.Core) NuGet package. Azure Cosmos DB Emulator to develop locally.
• It does 3 things very well: partitioning, replication, query/index.
• Hierarchy: account => database (grouping of containers and user/security permissions info) => container (like collections, graphs or tables) => item
• This is according to the type of API/syntax you’re going to use to access the data: Mongo DB, Graph API, Table Storage, DocumentDB, etc.
• It has predictable performances: you select how many operations/sec you need to have, and it scales up/down accordingly and automatically (no need to reserve CPU, memory, IO). “Request units”, or RU, are the actual “currency” of Cosmos DB. Required throughput is specified at the container level (in a multi-tenant it might be worth creating one container per tenant, so that you can scale them up differently and avoid spending too much if only a few require certain performances)
• Blazingly fast: guaranteed 10ms latency for reads and 15ms for writes…but there’s usually a single-digit latency (eg: 4–5ms for reads).
• Tunable consistency level (strong, eventual, session + 2 others), which are enabled with a click from the portal. More here, with sample scenarios for an e-commerce or social network.
• For session-based consistency, the session token can even be pulled out and used to override other requests, for example when you want session consistency across multiple clients.
• Partition keys is the key concept for Cosmos DB’s distribution and scalability. It’s a best practice to have a partition key with a large number of distinct values (e.g., hundreds or thousands). => many different logical partitions => can have more or less physical partitions according to required throughput (the number is changed dynamically by Cosmos itself). Having millions of logical partitions is not going to be expensive per se, because they can still correspond to very few physical partitions.
  It’s important to pick a property that allows writes to be distributed across various distinct values. So it’s important to pick a partition key that doesn’t result in “hot spots” within your application. Eg: using timestamps or dates is bad if you need to do a lot of writes, but good if you need frequent reads for timelines.
  Use 80/20 rule to optimise bulk of workload, understand ratio or writes/reads, pick a ley that is a common filter for queries.
  More about partitioning here.
• It has stored procedures, triggers and UDFs (user defined functions).
• A “change feed” is available to trigger API calls or other tasks when a document is inserted, updated or deleted. This could replace using Service Bus in some situations. More here.
• If an Azure region has an outage, automatic failover to an alternative region is enabled for the affected accounts (order of failover regions can be changed with drag & drop from portal, or customised in code for specific clients). When the outage is over, the affected accounts are recovered and a “conflict feed” is used to notify clients about write conflicts, so that a custom business logic (eg: first or last write wins, or anything else) is used to solve them. More here.
• Even for geographic distribution/replication, there just a single uri to access the database, which means the client apps don’t need to be re-deployed or re-configured if regions go up or down.
• Search for session THR2512 on the Microsoft Build Session Catalog to see how ASOS.com used Cosmos DB to support the peak of Black Friday. Also check out this other great video from Microsoft that talks a little bit about all this, and presents the ASOS use case for the recommendations engine (Spark, Event Hub, Data Lake, Data Factory and Service Fabric are also part of the overall solution)

Azure Storage

• Very reliable, data is replicated 3 times within the datacenter, and can also be replicated 3 more times on another datacenter.
• File Storage supports the SMB protocol, which makes it easy to move your file server to the cloud.
• Blob Storage is used for large (and small) unstructured files (audio, video or anything else really). Can store stuff in hot tier, cool tier (cheaper option, for data not accessed frequently, stored for at least 30) and archive tier (for data stored for at least 180 days — reading can take hours).
• Queue Storage, for small data, to be processed in a queue.
• File Storage is meant to be a remote disk accessible through SMB.
• Disk Storage is optimised for I/O operation, and used as a disk for VMs.
• Data is encrypted at rest. Support for authentication and authorization.
• Table Storage is a cheap and very fast nosql key-value store. No need to define a data schema (different rows can store different things). Can be used for caching.
• Data Lake Store for large relational/unstructured data, for when you don’t know the questions you’ll need to answer yet, and store data in its native format.
• SQL Warehouse Store for large relational data, for when you know the questions to be answered by data analytics.

Security

• Azure Active Directory can handle user registration, login, password management (eg: reset of forgotten password) etc.
• Azure Key Vault is a safe place to store passwords, connection strings, secret keys etc. Use it to avoid putting secrets in config files that are checked-in in source repos. But how to authenticate to Key Value? Use Managed Service Identity (MSI)

Other services

• Azure Redis Cache: in-memory, key-value storage
• Azure CDN, to replicate static files all over the world
• Azure Traffic Manager, to route users to locations with the lowest latency
• Azure Cognitive Services, for visual and text recognition
• Azure Machine Learning Studio
• Azure Bot Service
• Azure IoT Hub can ingest massive quantities of messages from IoT devices
• Azure IoT Edge allows some IoT devices to do calculations (even run AI models) locally without a dependency to the cloud
• Azure Data Factory for moving and transforming data
• Azure Analysis Services, Data Lake Analytics, Stream Analytics, Time Series Insights, Data Bricks are some of Azure services for data analysis
• Azure Media Services, to encode files and videos to multiple formats and resolutions, stich files together, generate thumbnails etc.

Messaging

• Azure Service Bus Topics and Queues help decouple services
• Azure Event Grid allows to subscribe to events like blobs being added
• Azure Event Hub can ingest massive amount of data, to be processed at a later time (with ELK for example). See this great article by Marco De Sanctis for a practical example that also uses Docker.

Monitoring

• Azure Application Insights: automatic for some things (eg: logging successful and failing requests, and how long they take to response), but add the SDK to your app for specific instrumentation and telemetry (eg: time sub-tasks inside a function, for example track how long the connection and the query to a DB takes).
• Azure Log Analytics plugs into any Azure service to gather diagnostics info.

Visual Studio and Tools

• If “Azure workload” is enabled in Visual Studio, it’s possible to work with Azure directly from VS.
  Cloud Explorer allows you to navigate through subscriptions and resources directly from Visual Studio, open a Azure SQL database, upload files to Azure Storage, attach the debugger to a App Service Web App or see its streaming logs and edit its config file.
  Specifically, the Snapshot Debugger allows you to debug an app in production without affecting its performances. When a “snappoint” is hit a snapshot is quickly taken with all the info about the call stack and the variables, and the app continues immediately. You can now inspect all that info and hopefully determine what’s going on.
  Publish wizard for many project types, and different profile files to deploy to different clouds, subscriptions, infrastructures (eg: local, Azure dev env, Azure production env).
• Azure Storage Explorer (Win, Mac, Linux) is a free standalone app that you can use to explore the content of many different types of storage: blobs, files, tables, queues, Cosmos DB, Data Lake.
• Azure CLI, to be used from your local machine or from the Azure Portal with Azure Cloud Shell, allows you to do everything you’d do from VS or the portal from the command-line or scripts.
• Visual Studio Team Services (VSTS) integrates with Git repos and allows to quickly set up build and deploy pipelines. It’s critical to iterate (develop, rest, release and start over) fast, so whatever you choose, do CI/CD.
• Visual Studio App Center: alternative to Bitrise or BuddyBuild for building Xamarin, iOS, Android and UWP apps on the cloud. It’s the evolution of HockeyApp, Xamarin Test Cloud and Code Push. It can connect to Git repos on GitHub, BitBucket and VSTS, and build the projects to create installable packages. Can run tests on 400+ unique devices. Can distribute builds to tests on a distribution list, and collect crash and usage info. Free for unlimited apps and distributions, but 1 build pipeline, 240 build minutes and max 30 minutes per build. Video here.

Who am I / what do I do? I proudly work as a Solutions Architect in the Mobile Team @ ASOS.com (iOS app | Android app), and we’re always looking for strong, friendly and talented developers that want to have an impact on how tens of millions of customers shop online. ASOS is the biggest online-only retailer in UK and, let’s be real, the best tech+fashion company in the world. Some of the technologies we use are Swift for iOS, Kotlin for Android, React and Node on the web frontend, .NET and Azure on the backend. If that sounds interesting to you, and you happen to live in beautiful London (or are willing to move here…after all it’s the best city in Europe except for some in Italy!), do get in touch with me!