Senior Databricks Platform Engineer at Sev1Tech

Arlington, Virginia, United States -

Full Time

Start Date

Immediate

Expiry Date

20 Feb, 26

Salary

0.0

Posted On

22 Nov, 25

Experience

5 year(s) or above

Remote Job

Yes

Telecommute

Yes

Sponsor Visa

Skills

Databricks, AWS, Data Lake, Apache Spark, Data Governance, Data Ingestion, ETL, Data Transformation, SQL, Python, Scala, Java, Data Security, Metadata Management, Cloud Platforms, Problem Solving, Communication

Industry

IT Services and IT Consulting

Description

Overview/ Job Responsibilities We are seeking a highly experienced and skilled Senior Data Lake Engineer to join our team. As the Senior Data Lake Engineer, you will play a critical role in establishing and configuring an enterprise-level Databricks solution to support our federal customer organization's data lake initiatives. This position offers a unique opportunity to work with cutting-edge technologies and shape the future of our federal customer's data infrastructure. If you are a highly skilled and experienced Senior Data Lake Engineer with expertise in Databricks and passion for building scalable and secure data lake solutions, we would like to hear from you. Responsibilities: Design, implement, and configure enterprise Data Lake solutions on AWS using Databricks, ensuring scalability, reliability, and performance. Work directly with cross-functional teams to gather requirements, analyze data integration needs, and translate them into actionable Data Lake architecture components. Build and configure Databricks workspaces, clusters, notebooks, jobs, and Delta Lake storage, integrating with AWS services such as S3, IAM, and KMS. Develop end-to-end data ingestion pipelines to extract, transform, and load data using Databricks, Apache Spark, and AWS services. Implement and maintain security controls including IAM policies, cluster security configurations, KMS-based encryption, and data masking to protect sensitive data. Create and optimize data transformation workflows, Delta Lake tables, and ETL/ELT processes to ensure high data quality and performance. Monitor and tune Databricks clusters and job workloads for reliability, cost optimization, and performance, leveraging autoscaling and workload management. Apply data governance best practices including cataloging, metadata management, and data lineage using Databricks Unity Catalog and AWS-native capabilities. Collaborate with cloud infrastructure teams to configure supporting AWS components such as S3 storage, networking, logging, monitoring, and access controls. Maintain detailed technical documentation including solution designs, data flow diagrams, configuration standards, and operational procedures. Stay up to date with advancements in Databricks, Delta Lake, Spark, and AWS services, integrating new features that improve automation and efficiency. Partner with data engineers, analysts, and scientists to implement data models and reusable transformation patterns within Databricks and AWS. Troubleshoot and resolve ingestion failures, cluster performance issues, pipeline errors, and data quality problems across AWS and Databricks environments. Ensure compliance with data governance, privacy, and security requirements by applying secure architecture patterns and validating controls throughout the data lifecycle. Support the evaluation and hands-on testing of new AWS and Databricks features or services that enhance the Data Lake environment. Minimum Qualifications Bachelor's degree in computer science, information technology, or a related field. Equivalent experience will also be considered. Proven experience in building and configuring enterprise-level data lake solutions using Databricks in an AWS or Azure environment. In-depth knowledge of Databricks architecture, including workspaces, clusters, storage, notebook development, and automation capabilities. Strong expertise in designing and implementing data ingestion pipelines, data transformations, and data quality processes using Databricks. Experience with big data technologies such as Apache Spark, Apache Hive, Delta Lake, and Hadoop. Solid understanding of data governance principles, data modeling, data cataloging, and metadata management. Hands-on experience with cloud platforms like AWS or Azure, including relevant services like S3, EMR, Glue, Data Factory, etc. Proficiency in SQL and one or more programming languages (Python, Scala, or Java) for data manipulation and transformation. Knowledge of data security and privacy best practices, including data access controls, encryption, and data masking techniques. Strong problem-solving and analytical skills, with the ability to identify and resolve complex data-related issues. Excellent interpersonal and communication skills, with the ability to collaborate effectively with technical and non-technical stakeholders. Experience in a senior or lead role, providing technical guidance and mentorship to junior team members. Relevant certifications such as Databricks Certified Developer or Databricks Certified Professional are highly desirable. Eligibility/Clearance Requirements: Must be eligible to obtain a Department of Homeland Security EOD clearance (Requirements 1. US Citizenship, 2. Favorable Background Investigation) Desired Qualifications Clearance Preference: Active DHS/CISA suitability - 1st priority Any DHS badge + DoD Top Secret - 2nd choice DoD Top Secret + willingness to obtain DHS/CISA suitability - 3rd choice (it can take 10-60 days to obtain suitability – work can only begin once suitability is fully adjudicated). About Sev1Tech LLC Founded in 2010, Sev1Tech provides IT, engineering, and program management solutions delivery. Sev1Tech focuses on providing program and IT support services to critical missions across Federal and Commercial Clients. Our Mission is to Build better companies. Enable better government. Protect our nation. Build better humans across the country. Join the Sev1Tech family where you can achieve great accomplishments while fostering a satisfying and rewarding career progression. Please apply directly through the website at: https://careers-sev1tech.icims.com/#joinSev1tech For any additional questions or to submit any referrals, please contact: Caitlin.maupin@sev1tech.com Sev1Tech is an Equal Opportunity and Affirmative Action Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, pregnancy, sexual orientation, gender identity, national origin, age, protected veteran status, or disability status.

Responsibilities

Design and implement enterprise Data Lake solutions on AWS using Databricks, ensuring scalability and performance. Collaborate with cross-functional teams to gather requirements and develop data ingestion pipelines.