What’s your strategy on cloud network logging and monitoring? Where do you keep vpc/vnet flow logs? How long do you store them for online investigation? What process do you follow to bring logs from cold storage and make them available online for investigation?
If you are using one of the cloud monitoring SaaS solution, they are likely to support capturing flow logs and make them available online for 30-45 days. If you are pushing the logs to your choice of SIEM, you can decide the number of days to keep data online as long as data growth does not slow down query performance.
Now, let’s come to the point! How effectively do you handle logs stored in archive or cloud storage (S3 or Azure Storage)? Is that something you can request from cloud monitoring partner and they provide the service for small fee? The chances are, they don’t have the capability and they are working on it! Luckily, you are already storing the flow logs for years for compliance reason and data is stored in cloud. Wouldn’t it be better if you build a tool to automatically retrieve data and make it a available for investigation?
Let’s take the case for Azure. You stored the flow logs in Azure Storage accounts and you can store up to 500 TB in one storage account. You know the resource group and the dates to pull data. It’s not hard to query and find list of blobs stored for your search filter. Once you know blob names and paths, you can download them easily.
For the purpose of this post, I created a PoC in C# + Azure Storage SDK. It’s very simple- one of the method will give you list of blobs for the specified path and another method will allow you to download a blob given it’s path & name.
GetListofBlobs:
Your goal is to specify the path as close to nearest day and minute but you can start with a broader parent path and narrow down the path as you learn the sub paths. I am going to try path ends with y=2019/m=08/d=30!
https://api.aspnet4you.com/api/azurestorage/GetListofBlobs?path=resourceId=/SUBSCRIPTIONS/B63613A2-9FC8-47AD-A65C-E1D1EBA108BE/RESOURCEGROUPS/AZ-AKS-V2/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/APP-GATEWAY2-NSG/y=2019/m=08/d=30/&maxResults=10
For brevity, I am going to truncate the result and show the paths for last few hours. We are going to use this information to download the blobs in next step. Keep in mind, the date and hours reflected are UTC time (I am 5 hours behind in Dallas!).
[ "resourceId=/SUBSCRIPTIONS/B63613A2-9FC8-47AD-A65C-E1D1EBA108BE/RESOURCEGROUPS/AZ-AKS-V2/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/APP-GATEWAY2-NSG/y=2019/m=08/d=30/h=02/m=00/macAddress=0003FF786772/PT1H.json", "resourceId=/SUBSCRIPTIONS/B63613A2-9FC8-47AD-A65C-E1D1EBA108BE/RESOURCEGROUPS/AZ-AKS-V2/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/APP-GATEWAY2-NSG/y=2019/m=08/d=30/h=02/m=00/macAddress=0003FF9FD70E/PT1H.json", "resourceId=/SUBSCRIPTIONS/B63613A2-9FC8-47AD-A65C-E1D1EBA108BE/RESOURCEGROUPS/AZ-AKS-V2/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/APP-GATEWAY2-NSG/y=2019/m=08/d=30/h=03/m=00/macAddress=0003FF786772/PT1H.json", "resourceId=/SUBSCRIPTIONS/B63613A2-9FC8-47AD-A65C-E1D1EBA108BE/RESOURCEGROUPS/AZ-AKS-V2/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/APP-GATEWAY2-NSG/y=2019/m=08/d=30/h=03/m=00/macAddress=0003FF9FD70E/PT1H.json", "resourceId=/SUBSCRIPTIONS/B63613A2-9FC8-47AD-A65C-E1D1EBA108BE/RESOURCEGROUPS/AZ-AKS-V2/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/APP-GATEWAY2-NSG/y=2019/m=08/d=30/h=04/m=00/macAddress=0003FF786772/PT1H.json", "resourceId=/SUBSCRIPTIONS/B63613A2-9FC8-47AD-A65C-E1D1EBA108BE/RESOURCEGROUPS/AZ-AKS-V2/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/APP-GATEWAY2-NSG/y=2019/m=08/d=30/h=04/m=00/macAddress=0003FF9FD70E/PT1H.json" ]
DownloadBlockBlob
I took the last item from the result above to download the blob.
https://api.aspnet4you.com/api/azurestorage/DownloadBlockBlob?blobPath=resourceId=/SUBSCRIPTIONS/B63613A2-9FC8-47AD-A65C-E1D1EBA108BE/RESOURCEGROUPS/AZ-AKS-V2/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/APP-GATEWAY2-NSG/y=2019/m=08/d=30/h=04/m=00/macAddress=0003FF9FD70E/&blobName=PT1H.json
Let’s see how the result looks like!
{
"$id": "1361",
"records": [
{
"$id": "1584",
"time": "2019-08-30T04:36:27.0299333Z",
"systemId": "a2d65e3c-5930-4ae5-a434-1653ef1b6816",
"macAddress": "0003FF9FD70E",
"category": "NetworkSecurityGroupFlowEvent",
"resourceId": "/SUBSCRIPTIONS/B63613A2-9FC8-47AD-A65C-E1D1EBA108BE/RESOURCEGROUPS/AZ-AKS-V2/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/APP-GATEWAY2-NSG",
"operationName": "NetworkSecurityGroupFlowEvents",
"properties": {
"$id": "1585",
"Version": 1,
"flows": [
{
"$id": "1586",
"rule": "UserRule_InternetToAppGateway",
"flows": [
{
"$id": "1587",
"mac": "0003FF9FD70E",
"flowTuples": [
"1567139763,65.36.92.217,172.240.16.5,8835,443,T,I,A"
]
}
]
},
{
"$id": "1588",
"rule": "DefaultRule_AllowVnetOutBound",
"flows": [
{
"$id": "1589",
"mac": "0003FF9FD70E",
"flowTuples": [
"1567139750,172.240.16.5,172.240.0.88,25404,443,T,O,A",
"1567139763,172.240.16.5,172.240.0.88,25405,443,T,O,A"
]
}
]
},
{
"$id": "1590",
"rule": "DefaultRule_AllowInternetOutBound",
"flows": [
{
"$id": "1591",
"mac": "0003FF9FD70E",
"flowTuples": [
"1567139737,172.240.16.5,52.239.154.134,25403,443,T,O,A"
]
}
]
}
]
}
},
{
"$id": "1612",
"time": "2019-08-30T04:40:27.1234683Z",
"systemId": "a2d65e3c-5930-4ae5-a434-1653ef1b6816",
"macAddress": "0003FF9FD70E",
"category": "NetworkSecurityGroupFlowEvent",
"resourceId": "/SUBSCRIPTIONS/B63613A2-9FC8-47AD-A65C-E1D1EBA108BE/RESOURCEGROUPS/AZ-AKS-V2/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/APP-GATEWAY2-NSG",
"operationName": "NetworkSecurityGroupFlowEvents",
"properties": {
"$id": "1613",
"Version": 1,
"flows": [
{
"$id": "1614",
"rule": "DefaultRule_AllowVnetOutBound",
"flows": [
{
"$id": "1615",
"mac": "0003FF9FD70E",
"flowTuples": [
"1567139990,172.240.16.5,172.240.0.88,25420,443,T,O,A",
"1567140020,172.240.16.5,172.240.0.88,25423,443,T,O,A",
"1567140020,172.240.16.5,172.240.0.88,25423,443,T,O,A"
]
}
]
},
{
"$id": "1616",
"rule": "DefaultRule_AllowInternetOutBound",
"flows": [
{
"$id": "1617",
"mac": "0003FF9FD70E",
"flowTuples": [
"1567140013,172.240.16.5,40.117.147.74,25421,443,T,O,A",
"1567140018,172.240.16.5,52.165.27.187,25422,443,T,O,A"
]
}
]
}
]
}
}
]
}
It would be injustice to the audience if I didn’t show some codes! Keep in mind, this code may not be optimized for performance but you will get the ides.
GetListofBlobs:
internal string[] GetListofBlobs(string path, int? maxResults)
{
List<string> list = new List<string>();
cloudBlobClient = csa.CreateCloudBlobClient();
CloudBlobContainer cbContainer = cloudBlobClient.GetContainerReference(flowLogContainer);
CloudBlobDirectory cbDirectory = cbContainer.GetDirectoryReference(path);
BlobContinuationToken blobContinuationToken = null;
do
{
var results = cbDirectory.ListBlobsSegmentedAsync(true, BlobListingDetails.Metadata, maxResults??25, blobContinuationToken,null,null).GetAwaiter().GetResult();
// Get the value of the continuation token returned by the listing call.
blobContinuationToken = results.ContinuationToken;
foreach (IListBlobItem item in results.Results)
{
list.Add((item as CloudBlockBlob).Name);
}
} while (blobContinuationToken != null); // Loop while the continuation token is not null.
return list.ToArray();
}
DownloadBlockBlob:
internal NSGFlowLogs DownloadBlockBlob(string blobPath, string blobName)
{
cloudBlobClient = csa.CreateCloudBlobClient();
CloudBlobContainer cloudBlobContainer = GetCloudBlobContainer(flowLogContainer);
CloudBlobDirectory cbDirectory = GetDirectory(cloudBlobContainer, blobPath);
CloudBlockBlob cbBlob = cbDirectory.GetBlockBlobReference(blobName);
string blockBlob = cbBlob.DownloadTextAsync().GetAwaiter().GetResult();
NSGFlowLogs nsg = JsonConvert.DeserializeObject<NSGFlowLogs>(blockBlob);
return nsg;
}
You can read flow logs many different ways but this is to give you the context how easy is to pull the logs with simple codes which can be turned into tool. I deployed the solution as micro-service into Azure AKS and it is exposed via INGNX proxy (part of the same cluster). You can find the source code of this solution on GitHub at https://github.com/aspnet4you/AzureFlowLogs.