Databricks advanced developer with (performance tuning expertise)

Senior Databricks Developer/Data Engineer should be an expert responsible for leading the design, development, optimization, and deployment of large-scale, robust data solutions and data pipelines within the Databricks Lakehouse platform. This role requires technical leadership, advanced performance tuning expertise, and a strong understanding of cloud architecture and DevOps practices.

Key Responsibilities

• Design & Architecture: Lead the end-to-end design and implementation of scalable data pipelines and a medallion architecture (bronze, silver, gold layers) for structured and unstructured data.
• ETL/ELT Development: Architect and implement complex ETL/ELT processes to process and transform large-scale datasets from diverse sources using Databricks notebooks, Delta Lake, and Apache Spark (PySpark, Scala, or SQL).
• Performance Optimization: Optimize Spark jobs for performance and cost-efficiency using advanced techniques such as partitioning, caching, cluster configuration tuning, and troubleshooting bottlenecks.
• Data Governance & Security: Implement data security strategies, compliance standards, and governance frameworks like Unity Catalog, including managing access controls, encryption, and data lineage.
• DevOps & Automation: Champion code quality and implement DevOps (CI/CD) pipelines using tools like Azure DevOps, GitHub Actions, or Jenkins to automate deployment and management of data assets and infrastructure.
• Troubleshooting & Monitoring: Develop and implement robust monitoring, logging, and alerting systems for production-grade pipelines to proactively identify and resolve issues.

Required Technical Skills and Qualifications

• Extensive Experience: 10+ years of experience in data engineering, with 5+ years specifically in Databricks development and leadership roles.
• Programming Mastery:

Should have at least 5 years of experience in Optimize Spark jobs for performance and cost-efficiency using advanced techniques such as partitioning, caching, cluster configuration tuning, and troubleshooting bottlenecks.

Advanced deep knowledge in Scala, Python, and SQL for optimizing code and build high performing data pipelines on Databricks.

Minimum 5 years of experience in Scala, Spark and Databricks programming

Must have 5- 10 years of experience implement complex ETL/ELT processes to process and transform large-scale datasets from diverse sources using Databricks notebooks, Delta Lake, and Apache Spark (PySpark, Scala and SQL).

Must be able to design and develop CICD pipelines that can push the code to different development and prod environments preferably to Azure using git actions

Must have 5 years of experience in Logging, Monitoring and Alerting systems.

• Big Data Technologies: Deep knowledge of the Apache Spark ecosystem, Delta Lake, and Lakehouse architecture.

Should have 2-3 experience in converting requirement to architecture and design in Spark and Databricks.

5 years of experience in Debugging and identifying performance in Databricks.

• Cloud Platform Expertise: Hands-on experience with Azure and their relevant data services (e.g., Azure Data Lake Storage, Azure Data Factory).
• Certifications: Professional certifications in Databricks (e.g., Databricks Certified Data Engineer Professional) or cloud data platforms are highly beneficial.

Working arrangements:

We strive to provide flexibility wherever possible. Based on this role’s business requirements, this is a hybrid position at Fort Worth TX(3 days a week) ,open to qualified applicants in the United States. Regardless of your working arrangement, we are here to support a healthy work-life balance though our various wellbeing programs.

The working arrangements for this role are accurate as of the date of posting. This may change based on the project you’re engaged in, as well as business and client requirements. Rest assured; we will always be clear about role expectations.

Work Authorization:
Please note, this role is not able to offer visa transfer or sponsorship now or in the future.

Salary and Other Compensation:

The annual salary for this position is depending on experience and other qualifications of the successful candidate.

Benefits: Cognizant offers the following benefits for this position, subject to applicable eligibility requirements:

• Medical/Dental/Vision/Life Insurance
• Paid holidays plus Paid Time Off
• 401(k) plan and contributions
• Long-term/Short-term Disability
• Paid Parental Leave
• Employee Stock Purchase Plan

Disclaimer: The salary, other compensation, and benefits information is accurate as of the date of this posting. Cognizant reserves the right to modify this information at any time, subject to applicable law.

The Cognizant community:
We are a high caliber team who appreciate and support one another. Our people uphold an energetic, collaborative and inclusive workplace where everyone can thrive.

• Cognizant is a global community with more than 300,000 associates around the world.
• We don’t just dream of a better way – we make it happen.
• We take care of our people, clients, company, communities and climate by doing what’s right.
• We foster an innovative environment where you can build the career path that’s right for you.

About us:
Cognizant is one of the world's leading professional services companies, transforming clients' business, operating, and technology models for the digital era. Our unique industry-based, consultative approach helps clients envision, build, and run more innovative and efficient businesses. Headquartered in the U.S., Cognizant (a member of the NASDAQ-100 and one of Forbes World’s Best Employers 2025) is consistently listed among the most admired companies in the world. Learn how Cognizant helps clients lead with digital at www.cognizant.com

Cognizant is an equal opportunity employer. Your application and candidacy will not be considered based on race, color, sex, religion, creed, sexual orientation, gender identity, national origin, disability, genetic information, pregnancy, veteran status or any other characteristic protected by federal, state or local laws.

Disclaimer:
Compensation information is accurate as of the date of this posting. Cognizant reserves the right to modify this information at any time, subject to applicable law.

Applicants may be required to attend interviews in person or by video conference. In addition, candidates may be required to present their current state or government issued ID during each interview.