Real-time wire data streams are also important sources of data for business and operational intelligence teams. In these types of scenarios, wire data is used to measure order transactions for real-time reporting on transaction volume, success, and failure rates; tracking patient admission rates at hospitals; as well as reporting on the weights and measures of airplanes prior to take-off.
Traditional methods of capturing and analyzing wire data include offline network packet analyzers. Newer approaches receive a copy of network traffic from a port mirror (SPAN) or network tap and reassemble those packets into full per-client sessions and transaction streams, analyzing the entire transaction payload in real time and generating metadata on those transactions without storing the actual packets.[2]
ExtraHop sensors deployed locally in data centers, clouds, and remote sites decrypt and process network data, extracting records and de-identified metadata which are sent securely to Reveal(x) 360 for behavioral analysis, real-time threat detection, and investigation.
ExtraHop's real-time stream processor transforms unstructured network packets into structured wire data at line-rate. Depending on the type of traffic, protocols involved, and security policies applied, a variety of processes are performed in real time:
The real-time stream processor decodes 70+ protocols (skip to Protocols We Decode) to understand, define, and act on that protocol's unique application boundaries in hybrid and cloud IT environments. This allows the processor to construct complete flows, sessions, and transactions for total application fluency, which in turn allows for higher-order content analysis through full-stream reassembly into wire data (derived from the wire protocol itself). The processor will even automatically resynchronize and recover in the event of traffic anomalies such as microbursts, that might otherwise result in packet loss.
Full-content analysis supports dozens of protocols, providing key performance indicators such as database methods used, file access by user, storage errors, DNS records and errors, web URI processing time and status codes and SSL certificates with expiration. The stream processor also captures sophisticated network metrics such as receive-window throttles, retransmission timeouts, and Nagle delays.
An event-driven programmable interface enables you to customize the telemetry captured by the stream processor. You can programmatically extract wire data events and threat indicators that are specific to your organization.
With this Application Inspection Triggers functionality, you can be as surgical or as verbose as necessary to extract anything from a simple header to the full application payload. For example, this data can include specific HTTP or SQL query strings used in injection attacks.
The same principle and functionality holds true for all of our natively decoded protocols. You can also use triggers to extract, measure, and visualize data from defined fields, or to decode proprietary protocols based on TCP and UDP.
Only de-identified metadata is sent to ExtraHop's cloud-based machine learning service, meaning that no payloads, filenames, strings, or other data categories that might contain sensitive data will leave your protected environment. ExtraHop has received SOC 2, Type 1 compliance certification for our machine learning technology, which you can learn about on our Compliance page.
From a SecOps perspective, a cloud-based record store provides fully hosted and managed search capability for streamlined incident investigation. Security teams can also leverage index record search and query of data from every segment of the hybrid environment for 360-degree visibility and situational intelligence.
One of the most challenging aspects of real-time analytics at cloud scale is, well, the scale itself. At ExtraHop, we do our best to make this easy for you as a user to parse the immense wealth of information that is wire data and derive meaningful insights no matter which perspective you're coming from.
ExtraHop enables you to create bespoke dashboards quickly. Drag-and-drop functionality enables you to build dashboards with widgets; if you want to create your own widget, all you have to do is select your desired data source and metrics, pick a visualization type, and save it to your dashboard of choice. You can quickly and easily export charts and background data points to PDF, Excel, or CSV.
Along with traditional methods of data visualization like charts and graphs, ExtraHop uses live activity maps to present a dynamic and intuitive view of your environment. Instead of manually creating and updating network diagrams as your IT environment changes, you can use live activity maps to visualize protocol-based connections between devices and applications in real time.
Database: RDBMSs: Oracle, Microsoft SQL Server, MySQL, PostgreSQL, Informix, Sybase, and DB2. NoSQL databases: MongoDB, Memcached, Redis, Riak. Metadata extracted include transaction timing, table/user access patterns, query errors, SQL queries and responses, and system-level commands.
Identity and Access Management: Active Directory visibility, including NTLM, Kerberos, LDAP, MSRPC, WINRM, SMBv3, and DNS monitoring for privileged identities and service accounts allows you to improve detection and facilitate audits. Reveal(x) extracts metadata including user/computer account activity, invalid or expired passwords, new privileged access, privileged access errors, DNS SRV lookups, LDAP binds, plain-text HTTP authentications, unknown SPNs, and forged Kerberos ticket detection.
Web Transactions: Full HTTP payload analysis of user activity, SOAP/XML, JSON, Javascript, APIs, etc. Extracted metadata includes URI, query parameters, host headers, and user agent, among others.
As you can see, Zeek reads in the PCAP, then extracts out metadata and puts them into TSV (tab separated value) formatted .log files labeled by type. Below is an example of the http.log in raw text form:
Now, if you recall, I talked about how wire data is the "metadata" of network traffic. In the example above, we have historical network traffic in the form of a PCAP. After Zeek reads the PCAP and outputs it to log, the information in the PCAP reduces from 7.4M to 256K. In other words, the "metadata" is only 3% of the captured network traffic (PCAP) but has nearly everything that you would want to use for network defense or hunting! Now you might not always have such an impressive reduction of size, but full network traffic (on the wire or saved in PCAP) will still be greater in size than wire data, whether it be Zeek, NetFlow or Splunk Stream.
Zeek is the wire data generator formally known as Bro (or even more widely known as Bro IDS). Since Bro was known as Bro IDS for many years, there's a misconception that Zeek is just another Snort. In fact, Zeek is less of an IDS than a network scripting language; at its base level, it can generate metadata network traffic (either from a live feed on the network OR in PCAP form).
Well obviously, you could set it up yourself! If you try Zeek, Splunk has written the Splunk Add-on for Zeek aka Bro that allows you to ingest the data Zeek creates easily. However, perhaps a more natural route for Splunk customers is Splunk Stream. As previously discussed, this is a free tool from Splunk that will generate network wire data. Check out these blogs on Splunk Stream installation and configuration! Once you have Stream or Zeek up and running, you may want to try out the new Splunk App, Splunk Essentials for Wire Data.
I hope this post has helped you understand the value of wire data and what it can help you find in your network. Many of the blogs in the Hunting with Splunk" blog series depend upon wire data for advanced hunting! The more you use wire data, the more you'll understand why it is the favorite data source of Security Splunkers like myself.
SecOps professionals have long considered endpoints and log files an integral piece of the cybersecurity puzzle. While it is true that endpoint data can provide valuable insight across multiple connected devices, these tools are limited in scope and vulnerable to breaches.
Wire data is the decoded raw information collected from computer and telecommunication network communications. This data can fall into two main categories: verbose (for instance, packet capture) and metadata, which is more commonly used for network security analysis.
No matter where the metadata originates, the information is focused on network traffic. That traffic can include live network traffic via network taps, switches and routers, or even localhost network interfaces. Security analysts study metadata to gain context into security events, detect unusual events, and pivot across proprietary data sources.
Metadata can capture more than 90 percent more of the usable data than a full packet system can capture. The ability to store and analyze metadata in real-time introduces predictability into network security protocol, allowing teams to discover threats that would have gone undedicated through packet capture.
caa09b180b