Azure Batch is considered as one of the High-Performance Computing options in Microsoft Azure. As the name says Batch – its couple of tasks combined & working independently in parallel. Its perfect when you need a huge computing power for a short duration.
When we try to run a high critical batch job we would take into considerations – No of VMs (resources) needed to run this job, queuing mechanism, scaling, monitoring, cleanup job etc. Azure Batch services help us to achieve these tasks helping us to manage and create pool of nodes. Later we can schedule it to run different jobs.
Azure Batch can automatically scale the infrastructure from a standalone node to hundreds of virtual machines nodes quickly on demand or based on the scheduled date time.
A Test team who can perform multiple tests for an application in parallel (100’s of virtual machines is spun-up on demand for testing).
The 3 important aspects of Azure Batch Service.
- Pool Start Task (Prepares node on the pool)
- Job preparation task (Helps us to prepare running of Job)
Let’s login to the Azure portal and create a New Batch account. Follow the steps as below. I forgot to link my Azure Batch account to a storage account. I navigated to my settings tab and linked a storage account to my batch service.
DEMO – A .NET application that uses Batch to process media files in parallel with the ffmpeg open-source tool.
I have downloaded the code from github to perform the demo. After downloading extract all the files and open & build the code in Visual 2017.
- Azure Batch account and linked general-purpose Azure Storage account
- Visual Studio 2017
- Windows 64-bit version of ffmpeg 3.4
- Add ffmpeg as an application package to your Batch account (Application Id: ffmpeg, Version: 3.4)
The below code will help us to process these media files and move it from IN storage folder to OUT storage folder (Independently & parallely).
After completing the above steps navigate to your .NET code and add the batch account credentials and storage account credentials.
// Batch account credentials
// Storage account credentials
After all the changes has been implemented we are good to go. Press RUN. It will 1st upload the files to the IN folder and later move these files in parallel to the OUT folder.
It created a Pool (WinFFmegPool) with 5 low priority nodes.
To Summarize, the workflow –
- We prepared the Azure Batch Account and linked it with the storage account (resources)
- We create a Pool of VM’s (Using .NET code)
- We create a Job and associate it with a Pool (Using .NET code)
- Added 5 tasks to the Job (These tasks are queued to run on its nodes, i.e. each task runs individually on its individual node)