We are seeking a Senior Azure Databricks Data Engineer to design, build, and optimize enterprise-scale data solutions that support analytics, operational reporting, and emerging AI initiatives. This role will be responsible for developing scalable ELT/ETL pipelines, modernizing legacy data environments, and delivering high-quality, cloud-native data products using Azure and Databricks technologies.

The ideal candidate combines strong hands-on expertise in Databricks, Spark, SQL, and Python with experience migrating legacy SQL-based workloads to modern cloud architectures. This individual will play a key role in advancing our enterprise data platform and enabling data-driven decision making across the organization.

Key Responsibilities

• Design, develop, and maintain scalable ELT/ETL pipelines for enterprise analytics and operational reporting.
• Build and optimize data workflows using Databricks, Apache Spark, SQL, and Python.
• Ingest, transform, and integrate structured and semi-structured data from multiple enterprise systems.
• Develop and maintain Delta Lake tables and Medallion Architecture (Bronze, Silver, Gold) data models.
• Lead migration initiatives from legacy/on-premises SQL environments to Azure Databricks and cloud-native platforms.
• Implement data quality, monitoring, observability, and reliability best practices.
• Collaborate with business intelligence, analytics, and data science teams to deliver trusted and curated datasets.
• Support AI/ML and GenAI initiatives by preparing and structuring data for downstream applications, including LLM and RAG-based solutions.
• Troubleshoot and optimize large-scale Spark processing jobs for performance and cost efficiency.
• Document data pipelines, architecture designs, data flows, and operational procedures.
• Participate in code reviews and contribute to engineering standards, best practices, and governance processes.
• Support production deployments and ongoing operational excellence through CI/CD and DevOps practices.

Job Qualifications

• Bachelor's degree in Computer Science, Information Systems, Engineering, or related field, or equivalent experience.
• 5+ years of experience in Data Engineering, Data Platform Engineering, or related technical roles.
• Strong hands-on experience with:
• • Azure Databricks
  • Apache Spark
  • SQL
  • Python
• Experience designing and maintaining enterprise ETL/ELT pipelines.
• Experience with Delta Lake and Medallion Architecture frameworks.
• Proven experience migrating workloads from SQL Server or other legacy data platforms to cloud-based solutions.
• Experience with data orchestration tools such as:
• • Azure Data Factory (ADF)
  • Apache Airflow
  • Databricks Workflows
• Strong understanding of data modeling, data warehousing, and large-scale data processing principles.
• Experience with Git-based source control and collaborative development workflows.
• Knowledge of CI/CD, DevOps, and automated deployment practices.

Preferred Qualifications

• Experience with Azure cloud services including Data Factory, Data Lake Storage, Synapse, Azure Functions, and related services.
• Experience supporting analytics environments within:
• • Gaming
  • Hospitality
  • FinTech
  • Highly regulated industries
• Exposure to AI/ML pipelines and MLOps practices.
• Experience with:
• • Vector databases
  • Retrieval-Augmented Generation (RAG)
  • Large Language Model (LLM) integrations
  • Generative AI solutions
• Knowledge of data governance, security, compliance, and observability frameworks.
• Experience tuning Spark workloads for performance and scalability.

Technical Environment

• Azure Cloud
• Azure Databricks
• Apache Spark
• Delta Lake
• SQL Server
• Python
• Azure Data Factory (ADF)
• Airflow
• Databricks Workflows
• Git
• CI/CD Pipelines
• DevOps Tools
• AI/ML & GenAI Data Platforms

By providing your phone number, you consent to: (1) receive automated text messages and calls from the Judge Group, Inc. and its affiliates (collectively “Judge”) to such phone number regarding job opportunities, your job application, and for other related purposes. Message & data rates apply and message frequency may vary. Consistent with Judge's Privacy Policy, information obtained from your consent will not be shared with third parties for marketing/promotional purposes. Reply STOP to opt out of receiving telephone calls and text messages from Judge and HELP for help.