Splunk Systems Integration Engineer

The Digital Modernization Sector has an opening for a highly skilled Splunk Systems Integration Engineer to join our team supporting our Unified Cyber Situational Awareness (UCSA) Splunk environment in Pensacola, FL or Columbus, OH. Under general supervision, defines and executes integration engineering activities within a project. These activities may consist of concept exploration and assessment, systems integration, GIS integration, legacy systems integration, performance management, technology assessment, testing and validation, and development and staffing of a systems integration plan. Possesses understanding of Splunk and its various components including forwarders, indexers, and web interfaces and has proficiency supporting Splunk on Linux in the AWS Cloud and on-site environment.

Primary Responsibilities:

• Develop and optimize complex queries and searches in Splunk to provide insights and support decision-making processes.
• Create custom dashboards and reports that align with contract objectives, security monitoring, compliance efforts, and ensure data is presented in a clear and actionable format.
• Work with cross-functional teams to understand their data needs and translate them into effective Splunk queries and reports.
• Analyze and troubleshoot Splunk queries to identify performance issues and optimize for faster execution.
• Develop and maintain efficient Splunk search strategies and reporting solutions to support monitoring, compliance, and security initiatives.
• Participate in designing and implementing data collection strategies, ensuring the integrity and accuracy of the ingested data.
• Maintain a strong understanding of Splunk best practices and continuously improve query performance and reporting quality.
• Performs concept exploration and assessment, systems integration, systems of systems integration, performance management, technology assessment, testing and validation.
• Analyzes and develops technical documentation detailing the integration and system performance.
• Design and Architecture: Design, implement, and maintain Splunk environments including forwarders, indexers, search heads, and deployment servers.
• Installation and Configuration: Install, configure, and integrate Splunk and its components, ensuring optimal performance, scalability, and security.
• Security Hardening: Harden the Splunk environment to meet security compliance standards and best practices.
• Monitoring and Maintenance: Monitor Splunk infrastructure health, performance, and capacity; perform routine maintenance tasks to ensure uninterrupted service.
• Troubleshooting: Troubleshoot issues related to Splunk infrastructure, including performance bottlenecks, data ingestion problems, and search optimization.
• Documentation: Create and maintain comprehensive documentation including architecture diagrams, installation guides, and troubleshooting procedures.
• Collaboration: Collaborate with cross-functional teams including security, network, and system administrators to ensure seamless integration of Splunk within the IT infrastructure.
• May test implementation of Internet Protocol version 6 (IPv6).
• May perform other duties as assigned.

Qualifications:

• Requires BS and 4+ years of prior relevant experience or Masters with 2+ years of prior relevant experience, additional years of experience will be accepted in lieu of a degree.
• 4+ years of hands-on experience working with Splunk, with a focus on query development, reporting, and dashboard creation.
• Must have a DoD 8570 IAT Level II (or Level III) certification (e.g. Sec+ CE).
• Must have a DoD Secret clearance or above.
• Proficient in Splunk Search Processing Language (SPL).
• Experience with Splunk reporting, alerting, and dashboard design.
• Ability to translate complex technical data into easy-to-understand reports and visualizations.
• Strong problem-solving skills with a focus on performance optimization and query tuning.
• Understanding of data onboarding and integration within Splunk environments is a plus.
• Proficient at data on-boarding activities including routing, parsing, and normalizing events to the Splunk Common Information Model (CIM).
• Proficient onboarding data using Splunk add-ons for Windows, Linux, and common third-party devices and applications.
• Experience onboarding data into Splunk via forwarder, scripted inputs, and modular inputs from a variety of sources.
• Excellent written and verbal communication skills, ability to work closely with multiple customers, manage expectations and track engagement scope.
• Experience with Splunk performing systems administration, including performing installation, configuration, monitoring system performance and availability, upgrades, and troubleshooting.
• General knowledge of networking and security troubleshooting (firewalls, routing, NAT, etc.).
• Splunk implementation and troubleshooting experience.
• Proficiency developing log ingestion and aggregation strategies per Splunk best practices.
• Perform integration activities to configure, connect, and pull data with 3rd party software APIs.
• Must have a Splunk Certified Architect certification.

Preferred:

• Splunk certifications, such as Splunk Certified Power User or Splunk Certified Admin.
• Eight (8) years of experience with Linux and Windows system administration or an advanced understanding of operating systems and common operating environments.
• Five (5) years of experience administering Splunk in distributed deployments.
• Security Skills—Knowledge of information assurance compliance and information security basics within CMS.
• Agile-based knowledge and skill, including experience with Scrum Ceremonies and work management tools (e.g., JIRA, Confluence).
• Certification as a Splunk Certified Architect or Splunk Certified Admin.
• Cloud certification (e.g., AWS Solutions Architect Associate, Azure Administrator).
• Ability to autonomously prioritize and successfully deliver results.