Satisnet Ltd, Suite B, Building 210, The Village, Butterfield Business Park, Great Marlings, Luton, Bedfordshire, LU2 8DL enquiry@satisnet.co.uk
+44 (0) 1582 369330

Utilising Zeek for Effective Context Enhancement within Your Azure Sentinel Instance

Friday 5th June 2020

Utilising Zeek for Effective Context Enhancement within Your Azure Sentinel Instance

To discover more about Zeek within your environment, please check out the precursor to this blog post – ‘Zeek Within Your Environment – Deployment and Set Up‘ – also co-written by two senior members of the Satisnet SOC Team, JJ Brown and Jawad Malik.

Your security operations team may be using Microsoft Azure Sentinel as a single pane of glass solution that maintains visibility of assets in both prem and the cloud. But are they considering what telemetry is most valuable for your business’ strong defensive posture and effective triage capabilities for your analysts?

That’s where the deployment of Zeek within your environment can help. We’ll be discussing this effective context enhancement solution further (following on from our blog on how to implement it within your environment)and how to connect Zeek to your Azure Sentinel instance.

Installing the Azure Log Analytics Agent

If your machine has connectivity to the internet, directly or through a proxy server, the following steps are recommended to download and onboard the Log Analytics Agent:

  1. In the Azure portal, click All Services found in the upper left-hand corner. In the list of resources, type Log Analytics. As you begin typing, the list filters based on your input. Select Log Analytics
  2. In your list of Log Analytics workspaces, select the workspace
  3. Select Advanced settings from the left hand pane
  4. Select Connected Sources, and then select Linux Servers
  5. Copy the command provided under “DOWNLOAD AND ONBOARD AGENT FOR LINUX”
  6. Run the command on the Zeek server, the command will download the agent, validate its checksum, and install it.

Configuring the Zeek Custom Logs in Azure Sentinel

Upload a text file with a sample of the Zeek conn log. You can find Zeek logs files located under /opt/zeek/logs/current/, but you may use the sample conn log below.

The wizard will parse and display the entries in this file for you to validate. Azure Monitor will use the delimiter you specify in order to identify each record. In our case, we used the default – ‘new line’.

{“ts”:”2020-05-11T18:58:09.065902Z”,”uid”:”CpQ1m43j5DlXZi5Iel”,”id.orig_h”:”192.168.0.1″,”id.orig_p”:59766,”id.resp_h”:”192.168.0.2″,”id.resp_p”:1900,”proto”:”udp”,”duration”:50.1710479259491,”orig_bytes”:2125,”resp_bytes”:0,”conn_state”:”S0″,”local_orig”:true,”local_resp”:false,”missed_bytes”:0,”history”:”D”,”orig_pkts”:17,”orig_ip_bytes”:2601,”resp_pkts”:0,”resp_ip_bytes”:0,”sensorname”:”zeek-virtual-machine”}

You can see how the steps look, in the following figures:

Zeek logs files are located under /opt/zeek/logs/current/. You can either provide a specific path and name for the log file as shown in the screenshot below, or you can specify a path with a wildcard for the name, for example, to collect all the logs, such as opt/zeek/logs/current/*.log.

Creating a KQL Function for Zeek Logs

The entire log entry will be stored in a single property called RawData. So, to separate the different pieces of information in each entry into individual properties, we’ll need to utilise the built-in parse_json() function

In the example below, the second line in the query will parse RawData field of the Zeek_DNS_CL table, return the value of the “ts” or timestamp json field and save the returned value in a dynamic property called timestamp_. The extend operator will add a new column called timestamp_ and return the results stored in the dynamic property. This is repeated for all the json elements in the RawData property.

KQL to Parse DNS Logs

  • Zeek_DNS_CL
    • extend timestamp_ = tostring(parse_json(RawData).ts)
  • uid_ = tostring(parse_json(RawData).uid), 
    • id_ = tostring(parse_json(RawData).id), 
  • protocol_ = tostring(parse_json(RawData).proto), 
    • trans_id = tostring(parse_json(RawData).trans_id), 
  • round_trip_time_ = tostring(parse_json(RawData).rtt), 
    • query_ = tostring(parse_json(RawData).query), 
  • qclass_ = tostring(parse_json(RawData).qclass), 
    • query_name_ = tostring(parse_json(RawData).query_name), 
  • qtype_ = tostring(parse_json(RawData).qtype), 
    • qtype_name_ = tostring(parse_json(RawData).qtype_name), 
  • response_code_ = tostring(parse_json(RawData).rcode), 
    • AA_ = tostring(parse_json(RawData).AA), 
  • RD_ = tostring(parse_json(RawData).RD), 
    • Z_ = tostring(parse_json(RawData).Z), 
  • answers_ = tostring(parse_json(RawData).answers), 
    • TTLs_ = tostring(parse_json(RawData).TTLs), 
  • rejected_ = tostring(parse_json(RawData).rejected), 
    • total_answers_ = tostring(parse_json(RawData).total_answers), 
  • total_replies_ = tostring(parse_json(RawData).total_replies), 
    • saw_query_ = tostring(parse_json(RawData).saw_query), 
  • saw_reply_ = tostring(parse_json(RawData).saw_reply), 
    • auth_ = tostring(parse_json(RawData).auth), 
  • addl_ = tostring(parse_json(RawData).addl) 

KQL to Parse HTTP Logs

  • Zeek_HTTP_CL
    • extend time_stamp_ = tostring(parse_json(RawData).ts), 
  • uid_ = tostring(parse_json(RawData).uid), 
    • id_ = tostring(parse_json(RawData).id), 
  • trans_depth_ = tostring(parse_json(RawData).trans_depth), 
    • method_ = tostring(parse_json(RawData).method), 
  • host_ = tostring(parse_json(RawData).host), 
    • uri_ = tostring(parse_json(RawData).uri), 
  • referrer_ = tostring(parse_json(RawData).referrer), 
    • version_ = tostring(parse_json(RawData).version), 
  • user_agent_ = tostring(parse_json(RawData).user_agent), 
    • origin_ = tostring(parse_json(RawData).origin), 
  • request_body_len_ = tostring(parse_json(RawData).request_body_len), 
    • response_body_len_ = tostring(parse_json(RawData).response_body_len), 
  • status_code_ = tostring(parse_json(RawData).status_code), 
    • status_msg_ = tostring(parse_json(RawData).status_msg), 
  • info_code_ = tostring(parse_json(RawData).info_code), 
    • info_msg_ = tostring(parse_json(RawData).info_msg), 
  • tags_ = tostring(parse_json(RawData).tags), 
    • username_ = tostring(parse_json(RawData).username), 
  • password_ = tostring(parse_json(RawData).password), 
    • capture_password_ = tostring(parse_json(RawData).capture_password), 
  • proxied_ = tostring(parse_json(RawData).proxied), 
    • range_request_ = tostring(parse_json(RawData).range_request), 
  • orig_fuids_ = tostring(parse_json(RawData).orig_fuids), 
    • orig_filenames_ = tostring(parse_json(RawData).orig_filenames), 
  • orig_mime_types_ = tostring(parse_json(RawData).orig_mime_types), 
    • resp_fuids_ = tostring(parse_json(RawData).resp_fuids), 
  • resp_filenames_ = tostring(parse_json(RawData).resp_filenames), 
    • resp_mime_types_ = tostring(parse_json(RawData).resp_mime_types), 
  • current_entity_ = tostring(parse_json(RawData).current_entity), 
    • orig_mime_depth_ = tostring(parse_json(RawData).orig_mime_depth), 
  • resp_mime_depth_ = tostring(parse_json(RawData).resp_mime_depth), 
    • client_header_names_ = tostring(parse_json(RawData).client_header_names), 
  • server_header_names_ = tostring(parse_json(RawData).server_header_names), 
    • omniture_ = tostring(parse_json(RawData).omniture), 
  • flash_version_ = tostring(parse_json(RawData).flash_version), 
    • cookie_vars_ = tostring(parse_json(RawData).cookie_vars), 
  • uri_vars_ = tostring(parse_json(RawData).uri_vars)   

Saving the query as a function will allow you to reuse the query, without having to retype out the query again and use it as a building block to expand or perform granular searches.

To create a KQL function for this query, we simply enter it into the query box in the Logs blade of Azure Sentinel and click the Save button.

Specify a name for the function, save as Function, and assign it an Alias which will be used to call the function.

To call the function, type the alias name and click Run.

Using Zeek Further

Here are some KQL queries to perform simple frequency and outlier analysis over the last 24 hours, using the KQL function above:

//Top Originators

  • Zeek_HTTP_Log                                 
    • where TimeGenerated > ago(24h)
      • summarize count() by host_ 
        • sort by count_ desc
          • limit 15

//Rare User Agents

  • Zeek_HTTP_Log
    • where TimeGenerated > ago(24h)
      • summarize count() by user_agent_
        • sort by count_ asc
          • limit 15

//Rare Host Headers

  • Zeek_HTTP_Log
    • where TimeGenerated > ago(24h)
      • summarize count() by host_
        • sort by count_ asc
          • limit 15

Conclusion

Hopefully, this article has helped you understand how easy it is to use Azure Sentinel to ingest Zeek data, deploy and onboard a Linux agent with a simple one-line command and create KQL searches and functions.

You can further develop and expand your Zeek hunting scenarios by leveraging the threat detection content, for example rules and queries, on SOC Prime’s Threat Detection Marketplace (TDM).

References

Zeek – Quick Start Guide
Zeek: – Installation/Installing
Microsoft Tech Community – Using KQL Functions to Speed Up Analysis in Azure Sentinel
Microsoft Tech Community – Tip: Easily Use JSON Fields in Sentinel
Microsoft Docs – Collect Custom Logs in Azure Monitor
SOC Prime – uncoder.io