Friday, May 19, 2017

Azure cost optimization – Send unassigned Azure public IP list using Azure PowerShell

Abstract


Cost of Microsoft cloud is operational. This means every month you are going to get bill just as your grocery/ electricity/ phone bills. Having said that, companies are struggling these days to perform cost optimization or cost reduction on their Microsoft Azure spending.
Current article provides an Azure cost optimization tip by sending the list of unassigned public IPs present in your Azure subscription; as an email using Azure PowerShell and Azure Automation runbook.
Also I am talking about cost optimization (or cost reduction) for Public IP addresses related to Azure Resource Manager (ARM) mode.

Knowledge Update for you!!


In case you are not aware, it is worth to mention that, Azure has 2 types of IP addresses –
  1.  Dynamic public IP address
  2.  Static public IP address
IP addresses are always attached to Network Interface Card (NIC) of azure virtual machine. So their cost model is as below.


Dynamic public IP address

  1. IP address is attached to NIC of Azure VM, and VM is running, you are charged for approx. 197 INR/ month or 3$/ month.
  2. IP address is attached to NIC of Azure VM, and VM is in Stopped(De-allocated) state, you are not charged for dynamic public IP address.
Static public IP address

  1. Azure Static Public IP addresses are charged for reservation and usage both. This is double cost that of azure dynamic public IP cost.
  2. First 5 static public IP addresses cost for reservation is FREE. Only charged for usage at 197 INR/month or 3$/ month.
  3. All additional static public IP addresses are charged for Usage and Reservation both, as below -
a.      IP address is attached to NIC of Azure VM, and VM is running; you are charged for approx. 197 INR/ month or 3$/ month for reservation and 197 INR/ month or 3$/ month for usage. Total 394 INR/ month or 6$/ month.
b.      IP address is attached to NIC of Azure VM, and VM is in Stopped(De-allocated) state, you are charged for approx. 197 INR/ month or 3$/ month for reservation. There will be no usage charges.
c.       IP address is created in Azure subscription, not attached to any resource, still you are charged for approx. 197 INR/ month or 3$/ month for reservation. There will be no usage charges.

How we are going to save the azure operational cost?

From the above knowledge paragraph, it is evident that we need to be alert for Static Public IP cost only. As dynamic public IPs are not charged is not being used.
Hence we can optimize/ reduce/ save Azure billing “by deleting Azure Public static IP which is reserved but not attached/ associated to any resource”.
I am going to give you the Azure Automation PowerShell runbook for sending emails of such static unused but reserved public IP addresses.
Hope we are clear here 😊.

Create Sendgrid account on Azure to send emails


To send emails on Azure I always prefer sendgrid as it provide almost 25000 email/ month free. Get started with email account creation from here - http://sanganakauthority.blogspot.in/2017/04/send-email-from-azure-sendgrid-using.html.

Create Azure Automation account and runbook


Below link specifies the steps to provision Azure Automation account – Create Azure Automation account.
After azure automation account creation, select option as Runbook -> Add a runbook -> Quick Create. Provide the name of runbook as “List-UnassignedPublicStaticIPs”. Runbook type as “PowerShell”. Provide meaningful description. Then click on Create.
Open newly created runbook and click on Edit option.
Now to run the PowerShell code in this runbook against our subscription, we need to provide authentication logic in the runbook first.  For the same add below code at the top of runbook –

$connectionName = "AzureRunAsConnection" #this should be your azure connection name. this can be retrieved from your automation account -> Assets -> Connections

try
{
    # Get the connection "AzureRunAsConnection "
    $servicePrincipalConnection = Get-AutomationConnection -Name $connectionName      
    "Logging in to Azure..."
    $account = Add-AzureRmAccount `
        -ServicePrincipal `
        -TenantId $servicePrincipalConnection.TenantId `
        -ApplicationId $servicePrincipalConnection.ApplicationId `
        -CertificateThumbprint $servicePrincipalConnection.CertificateThumbprint 
}
catch {
    if (!$servicePrincipalConnection)
    {
        $ErrorMessage = "Connection $connectionName not found."
        throw $ErrorMessage
    } else{
        Write-Error -Message $_.Exception
        throw $_.Exception
    }
}
Write-Output $account

The above code segment ensures that your azure automation runbook don’t run into famous error “Run Login-AzureRmAccount for login”.

Retrieving unassigned static public IPs in Azure PowerShell


If azure public IP is not attached to any resource then its IpConfigurationText property is always null. Also, the public IP type is dynamic or static can be retrieved from property PublicIpAllocationMethod.
So we will use these two properties as a filter to retrieve public static IPs which are not allocated to any resource and still getting charged.
So below is the full command to retrieve azure public static un-assigned public IPs which can be delete to reduce monthly azure cost.

$unassignedIPs = Get-AzureRmPublicIpAddress | Where-Object IpConfigurationText -EQ null | where-object PublicIpAllocationMethod -EQ "Static" | ConvertTo-Html Name, IPAddress, ResourceGroupName,PublicIpAllocationMethod | Out-String

$unassignedIPs will contain the IP addresses. Now we need to send this is email. Therefore use below code of Sendgrid PowerShell to send email -
$Username ="YourUserName"

$Password = ConvertTo-SecureString "YourPassword" -AsPlainText -Force

$credential = New-Object System.Management.Automation.PSCredential $Username, $Password

$SMTPServer = "smtp.sendgrid.net"

$EmailFrom = "No-reply@azureadmin.com"

[string[]]$EmailTo = "YourEmail"

$Subject = "List of Un-assigned Public IPs"

$Body = $unassignedIPs + "Remember, every un-assigned Static IP is charged at the rate of <b>200 INR/Month</b>. So please delete it if not required."

Send-MailMessage -smtpServer $SMTPServer -Credential $credential -Usessl -Port 587 -from $EmailFrom -to $EmailTo -subject $Subject -Body $Body -BodyAsHtml
Write-Output "Email sent succesfully."

This code will send email all unassigned static public IPs.

Now run the “Test Pane” option of automation runbook and verify that you receive an email about un-assigned static public IP addresses present in your subscription. Then publish the runbook attach a schedule to it so that this runbook will execute automatically and you will receive email. Best frequency would be every Monday morning 9AM when office starts 😊.

Next Step

Obviously, delete the unassigned Azure public IPs from your subscription. You have the list received in your inbox; now delete it manually. Or if you are smart enough, you can use the command –  Remove-AzureRmPublicIpAddress

That’s all folks.
Happy Cost Optimization on Cloud!!

Saturday, May 6, 2017

Run Login-AzureRmAccount to login

Abstract

Honestly, I did not know. This error has created so much of the frustration in the developers; who wishes to use Azure PowerShell and Azure Automation. This blog post is dedicated to solving the error “Run Login-AzureRmAccount to login”.

Background

Any Azure RM [a.k.a Azure Resource Manager] PowerShell command execution first requires authentication done against your Azure subscription. So if you fire any command without Login-AzureRMAccount; above sweet error comes.

Reproducing the error

Let’s first reproduce this error.

I am assuming you already Azure PowerShell module installed. If not refer here for installation steps. Now open PowerShell and run the command to retrieve all Azure VMs present in the Azure subscription –

Get-AzureRmVM

Error appears – “Run Login-AzureRmAccount to login”.

Solution is simple, run the command “Login-AzureRmAccount” and it opens up a pop up. Enter the credentials. After this run the command of retriving VMs again and everything works.

So, locally it’s easy to get rid of this error. How do we solve the error in Azure Automation account? Let’s first reproduce the same in Azure Automation account. I already have one Azure Automation account created as per the earlier blog post here. Refer section “Provision Azure Automation Account” in the blog post.

Click on Runbooks -> Add a Runbook. Give the name of your choice, select the type as “PowerShell”, and provide description of your choice. Then click on Create. After runbook is created on the Azure Portal, Open it by clicking on Edit option. Type the command as “Get-AzureRmVM”. Then to test the command click on “Test Pane” as highlighted below –



Click on Start button in Test Pane window to start the execution. There you receive the error again – “Run Login-AzureAccount to login”. Now here is the catch. Automation account runbooks runs in the background hence they can’t throw a pop up wherein you can put up your credentials. So how do we resolve it?


Solution is – Use Azure AD Service Principal

Service principal means you are treating an application as a user and giving full access to it so that it can perform any action against your azure subscription. As Azure subscription is always present in the Azure Active Directory tenant; we must add the information of our application in Azure AD tenant and this is nothing but the service principal.

So how do we create a Service principal? Well you don’t have to create because it already exists if you have an Azure ARM automation account created.

Open Assets -> Connections -> AzureRunAsConection. This shows type as Azure service principal and there are many Ids present as highlighted below –



Application Id is the one by which your Automation account is identified as Service principal in Azure AD. Tenant id is nothing but Azure AD id under which your subscription exists. Subscription Id is the actual Azure subscription Id.

Let’s verify this exist in your Azure AD as well. For the same, on Azure portal open Azure Active Directory -> App Registrations. You will see an Application Id same as what we have observed under automation account connection.



This means AzureRunAsConection of automation account is acting as Service principal. Hence it can be used for authentication against the subscription and also to perform operations against our azure subscription. With this let’s write some PowerShell code to perform authentication using service principal.

Authenticating using Service principal

Code for authenticating Azure Automation account runbook using Automation connection as Service principal is shown below –

$connectionName = "AzureRunAsConnection"
try
{
    # Get the connection "AzureRunAsConnection "
    $servicePrincipalConnection = Get-AutomationConnection -Name $connectionName      
    "Logging in to Azure..."
    $account = Add-AzureRmAccount `
        -ServicePrincipal `
        -TenantId $servicePrincipalConnection.TenantId `
        -ApplicationId $servicePrincipalConnection.ApplicationId `
        -CertificateThumbprint $servicePrincipalConnection.CertificateThumbprint 
}
catch {
    if (!$servicePrincipalConnection)
    {
        $ErrorMessage = "Connection $connectionName not found."
        throw $ErrorMessage
    } else{
        Write-Error -Message $_.Exception
        throw $_.Exception
    }
}
Write-Output $account

Add above code segment in any runbook you wish to in Azure Automation account and you will never receive error of “Run Login-AzureRMAccount to login”.

I did the same in my sample runbook and VM list received. Below is the output –



That’s all folks.

Happy error handling!!

Thursday, April 27, 2017

Send email from Azure SendGrid using PowerShell

Abstract

Sendgrid is popular. No doubt about it! Especially for Dev and Test scenarios where you get 25000 free emails a month. This post describes how can you use Sendgrid to send email using PowerShell.

Don’t expect me to write blog post on C# code for sending email using Sendgrid. Because, Sendgrid will keep updating their API (like V2, V3 and so on); ultimately C# code will also change. So you can refer C# or other languages code sample from SendGrid documentation itself.

Provision SendGrid account on Microsoft Azure

Let’s create a SendGrid account quickly on Azure. Login to Azure portal on the link – http://portal.azure.com . Sendgrid today available only in selected region. Example, not available in India region. I am based out of India so nearest data center for me is SouthEast Asia referred as SEA in the blog). I will select the SEA datacenter for the creation of SendGrid account on Azure. Therefore first let’s add Resource group in SEA region and then create SendGrid in the same Resource group.

Select “Resource Groups” in left had pane. It lists all resource groups present in your subscription. Click on “Add” button present at the top. Provide the resource group name as SendEmailRG, subscription of your choice and location as SEA as shown below. Click on Create to finish the resource group creation process.



Click on New. You will see a search box at the top. Type Sendgrid on the search box; intelligence will automatically display “Sendgrid Email Deliverys”. Select the same and press enter. Another horizontal blade will open. Select the Sendgrid Email Delivery option and click on Create.

Provide the name of account as “EmailSample”. Provide the password of your choice and remember it. We will need it later. Select the Azure subscription and select existing option for resource group. Select SendEmailRG resource group. This step will ensure that SendGrid account in created in SEA region.

On pricing tier; select the option as Free as shown below –



If you have money feel free to select the other paid pricing tiers.

Provide the contact information. I have provided below. Remember you will receive confirmation email from SendGrid. So don’t put values like below; rather give genuine values.



Select “I give…” checkmark for legal terms and click on Purchase. Then proceed ahead and finish the SendGrid account creation process.

Retrieve essential detail from SendGrid account

To send email we need username, password and SMTP server name. This information is available on Azure portal under your sendgrid account. So go to the SendGrid account we created in above step from Azure portal. Click on Configurations and you will see the required information listed as shown below –


PowerShell to Send Email

Let’s first define the essential parameters based on the information we captured from configurations tab of sendgrid tab –

$Username ="YourUserNameFromPortal"

$Password = ConvertTo-SecureString "YourPassowrdWhichYouHadEnteredDuringSendGridCreation" -AsPlainText -Force

$credential = New-Object System.Management.Automation.PSCredential $Username, $Password

$SMTPServer = "smtp.sendgrid.net"

Any email object requires necessary information like email from, email to, subject line. Let’s add this information as parameters –

$EmailFrom = "No-reply@azureadmin.com"

[string[]]$EmailTo = "YourEmail@gmail.com"
$Subject = "Sending sample email using SendGrid Azure and PowerShell"

$Body = "This is sample email sent using Sendgrid account create on Microsoft Azure. The script written is easy to use."

If you observe, I am using array for $Emailto variable. This is because if I want to add multiple email address into “To” list, I should be able to do it using array. So I want to send email to multiple addresses I will add them as below –

[string[]]$EmailTo = "YourEmail1@gmail.com", “YourEmail2@yahoo.com”

Then use below command to send the email –

Send-MailMessage -smtpServer $SMTPServer -Credential $credential -Usessl -Port 587 -from $EmailFrom -to $EmailTo -subject $Subject -Body $Body -BodyAsHtml

Manage option on SendGrid account

Look at the below screenshot for highlighted option.



This option will help you in generating api key, monitoring of your sendgrid account, failed emails , reports and so on. No this is not another blog post on another day because Sendgrid documentation is detailed to explain all the options. Why you want me to write blog post on the information which is already explained so well?

Note -

Sendgrid provides API Keys which can be used for providing limited access. The above PowerShell code do not use API Key for sending emails. In case you want to use SendGRid API Keys based access control and then send the email using PowerShell; I will suggest to write c# code exe with required parameters and then invoke it using Powershell.  😊


Happy Emailing!!

Tuesday, March 7, 2017

How to download Azure blob storage contents in Azure Linux VM using Azure CLI

Abstract

I always get this question – how can I download Azure blob storage files in Azure Linux VM? When I say use Azure CLI (Command Line Interface) then next question asked is – Do you have step by step guide?
Well, this blog post is the answer to both questions.

The high level approach is outlined below –
  1. Provision Linux Azure VM in a subnet. [Of course this step is out of scope of this article. For detailed steps refer - this guide .]
  2. Install Azure CLI in Linux VM
  3. level Upload sample files to azure storage and then download them in a folder in Linux VM.

This article assumes you understand Azure storage and related concepts.

Wow, this is first blog post from me on Linux and Azure.

Prepare you Azure Linux VM

I have provisioned an Azure Linux VM with OS as CentOs 7.2. Added this VM in a Subnet, with NSG having only port 22 inbound open. Also I have attached a public IP to this VM so that I can make SSH to this VM from anywhere over port no 22. Step by step guide link is already shared above.
If you are having different OS than CentOS then commands in below steps will change however high level approach remains same.

Install CLI in Azure CentOS VM

First make SSH to your Linux VM and run command “sudo su” [without double quotes]. So in subsequent steps we will not face awesome “access denied” or “permission denied” errors. Or we don’t have to add “sudo” word in every command we run.

There are two versions of Azure CLI –
1.0    – This stuff is written in node.js and supports both Classic [old way of doing things on Azure] and ARM [new fancy way of doing things on Azure].
2.0    – To make this version impressive Microsoft calls it “Next generation CLI” and is written in python. Only supports ARM mode.

I will be using 2.0 version. Hence I need Python as well installed on the Linux Azure VM. So let’s first install python latest version on Azure CentOS VM.
Let’s make sure that yum is up to date by running below command –

sudo yum -y update

-y flag tells system that “relax, we are aware that we are making changes, hence do not prompt for confirmation and save our valuable time”. This command execution will take good amount of time.
Next install yum-utils using below command –

sudo yum -y install yum-utils

Now we need install IUS (Inline with upstream stable). Don’t get scared by name. This is community project which will ensure that whatever version we install for Python 3, we will get the most stable version. Run below command to install IUS –
sudo yum -y install https://centos7.iuscommunity.org/ius-release.rpm

After IUS now we can install recent version of Python. As of writing, the recent version is 3.6 but I will install 3.5 to be on safer side. In python 3.5 version I see 3.5.3 is the latest so let’s install it.
sudo yum -y install python35u-3.5.3 python35u-pip

To verify, simple run below command and it’s output should be 3.5.3.
python3.5 -V

Now install the required prerequisites on CentOS using below command –
sudo yum check-update; sudo yum install -y gcc libffi-devel python-devel openssl-devel

Finally, back to installation of CLI 2.0 -


curl -L https://aka.ms/InstallAzureCli | bash
This may prompt you to download CLI in which directory. Press enter to keep the default path of installation which would be “/root/lib/azure-cli”. Similarly keep pressing enter if more prompts are displayed.
Restart command shell to take changes effect –
Exec -l $SHELL
Just type “az”[without quotes] and it should you Azure cli commands information in CentOs. This means installation of Azure CLI 2.0 on Linux is successful.

Run below command to list storage related commands.
az storage -h

Upload sample files to Azure Storage

This step is straight forward. Use Azure portal and create one standard [not premium] ARM based storage account. Create container and upload 4 sample files in the container. It would look like below –






Add Azure account to CLI

Run command  as shown below. It will prompt you with a code and link to enter the code. After this you will be asked for login using existing azure related credentials. Successful login will show you the subscriptions associated to your account as below –



Set storage account and download the blob

Now set credentials for storage account.
export AZURE_STORAGE_ACCOUNT=YourStorageAccount
export AZURE_STORAGE ACCESS_KEY=YourStorageAccountKey

Create a directory named as test1 using the command. This is the directory in which we will download blob contents. -
mkdir test1/

After this run below command to download the blob file in test1 folder
az storage blob download -c sample -n File1.txt -f /test1/File1.txt
az storage blob download -c sample  /test1

Change directory to test by command cd test1 and run ls -l. This should list File1.txt as shown below.

Limitation

Using Azure CLI you can’t download all the blob from a container. You have to download each and every blob individually; bulk download of azure blobs is not supported.

Resolution to Bulk download

To download all blobs from a container instead of Azure CLI, we will need to use Azure XPlat CLI. Or we can also use Powershell as it is open source now [although I have not tried yet]. It’s common to refer many approaches to achieve one task when you are in open source. J
Azure XPlat CLI is a project that provides cross platform command line interface to manage Azure. Refer documentation here - https://github.com/Azure/azure-xplat-cli. But this is another blog on another day.

Conclusion

So I hope now you understand how easy it is to download azure blob storage contents in Linux Virtual Machine.
Please provide your valuable comments. Good news is its free!!
Keep Downloading!!