Microsoft Sentinel is a cloud-native security information and event management (SIEM) platform that enables organizations to collect, analyze, and respond to security events across their enterprise (for an introduction please see the first post in this series). Data ingestion is a critical component of the Sentinel platform, as it allows organizations to collect and import data from various sources for storage, processing, and analysis.
In today’s digital age, organizations generate vast amounts of data daily, making it necessary to have a system that can handle the data’s volume and variety. With Sentinel’s data connectors, organizations can collect data from various sources and bring it into the system seamlessly. These data connectors play a vital role in data ingestion and are essential in making Sentinel’s data processing and analysis capabilities effective.
This blog post provides an introduction to data ingestion and data connectors within the Sentinel platform. By leveraging these capabilities, organizations can quickly collect and analyze security data to identify threats and vulnerabilities, ultimately enhancing their overall security posture.
Below you’ll find all parts of this blog series.
- The Sentinel Awakens: Microsoft Sentinel Introduction
- Feeding the Beast: Data Ingestion with Microsoft Sentinel
- Sentinel Teams Up: Integrations with Microsoft Teams
What is Data Ingestion?
Data ingestion is the process of collecting and importing data from various sources into a system for storage, processing, and analysis. The sources of data can be structured or unstructured, such as databases, applications, websites, sensors, and social media. The data is typically transformed and cleaned (normalized) during the ingestion process to ensure its quality and consistency. Once the data is ingested, it can be processed and analyzed to extract insights and make data-driven decisions.
What are Data Connectors?
Data connectors are software components that enable data ingestion from various sources into a system. They act as a bridge between the data source and the system, providing a standard interface to connect and extract data. Data connectors can be pre-built or custom-built, depending on the source of data and the system’s requirements. They can extract data from various sources, such as databases, applications, websites, and files, and convert them into a format that can be ingested into the system.
Benefits of Data Ingestion and Data Connectors
- Data-driven decisions: Data ingestion and data connectors enable organizations to collect and process data from various sources, providing a comprehensive view of their operations. This information can be used to make data-driven decisions and improve business performance.
- Efficiency: Data ingestion and data connectors automate the process of collecting and importing data, reducing the time and effort required to do so manually.
- Improved accuracy: Data connectors ensure data quality and consistency by standardizing the data format and cleaning it during the ingestion process.
- Scalability: Data ingestion and data connectors allow organizations to scale their data processing capabilities, enabling them to handle large volumes of data.
- Integration: Data connectors can integrate with various systems and platforms, providing a seamless experience for organizations and their users.
Let’s configure a data connector in Sentinel!
1. You’ll need to have access to a Sentinel workspace to get started. Please see the first blog post in this series in order to get started. From the Sentinel dashboard find Content Management > Content Hub as shown in the screenshot:
Note
– Ignore the Data connectors under Configuration for now as installing connectors that way is being retired soon.
2. Within the Content hub a variety of different solutions can be added into Sentinel consisting of analytics rules, workbooks, playbooks, and data connector(s). For the purposes of this demo we’ll the install Azure Active Directory (AAD) solution (to be used in a future demo):
3. Create the Solution:
4. Select the appropriate workspace and Review + Create and wait for the deployment to finish (this can take several minutes):
5. Return to the Content Hub and filter for ‘Installed’ status to find the solution and open the ‘Manage’ settings to finalize installation of the data connector:
6. The first item listed should be the data connector with a warning symbol requiring further configuration:
On the configuration page we can fine-tune exactly which logs to ingest with the ‘Sign-in logs’ and ‘Audit logs’ being the two most important ones for most organizations. For this demo all are selected:
The data connector status should go from ‘Not connected’ to ‘Connected’ within a few minutes time.
Note
If you are following along with this blog post series then be sure to repeat the same steps as above but for the ‘Azure Active Directory Identity Protection‘ data connector as it will be used in the next blog post.
Want to know more? We’re doing a thesis on Sentinel!
This Master thesis aims to address the need for a simplified approach to set up detection and automated response for common security threats on cloud and hybrid systems using Microsoft Sentinel. We are still looking for a few companies that are eager to contribute to this research with limited spots available.
Pre-requisites:
– Pre-existing SOC team consisting of at least 5 technical members
– Does not have to be a Danish company
Reach out to us if you’re interested in a collaboration with us. More information provided in the picture below. Contact information can be found in the side profiles.
Conclusion
Data ingestion and data connectors are essential components of a data-driven organization. They enable organizations to collect, process, and analyze data from various sources, providing insights that can improve business performance. Data connectors automate the process of collecting and importing data, reducing the time and effort required to do so manually. They ensure data quality and consistency, enabling organizations to make accurate decisions based on the data. Data ingestion and data connectors are scalable and can integrate with various systems, providing a seamless experience for organizations and their users. Organizations looking to harness the power of their data should consider implementing data ingestion and data connectors as part of their data strategy.
Stay tuned on this blog for the third part of this series where we’ll take a look at setting up a first alert and integrating it into Microsoft Teams!
Security consultant with focus on cloud and Azure.
