You will own and improve the engineering practices behind reliable, scalable enterprise data pipelines, with a strong focus on Apache Airflow, Astronomer, automation, and production reliability. This role combines data platform engineering, CI/CD, Python development, Azure Data Factory, SQL Server, observability, and DevOps practices, focusing on creating consistent standards for how data pipelines are developed, tested, deployed, monitored, and supported across multiple environments.

Key Responsibilities

• Own the technical implementation, administration, and operational support of Astronomer and production Apache Airflow environments.
• Design, develop, maintain, test, and troubleshoot Airflow DAGs supporting enterprise data workflows.
• Establish engineering standards for DAG structure, naming conventions, dependencies, scheduling, retries, alerting, error handling, and recovery.
• Build and maintain automated CI/CD pipelines for Airflow DAGs, Python packages, plugins, configurations, and supporting platform components.
• Automate pipeline deployments and promotions across development, QA, UAT, and production environments.
• Implement automated DAG validation, unit testing, integration testing, SQL validation, and deployment checks to improve release quality.
• Establish Git-based development practices covering source control, branching, pull requests, code reviews, release management, artifact versioning, and rollback procedures.
• Manage Airflow connections, variables, secrets, pools, queues, executors, dependencies, and environment configurations.
• Integrate Airflow with Azure Data Factory, Microsoft SQL Server, REST APIs, file systems, cloud storage, and enterprise applications.
• Develop and support Azure Data Factory pipelines, datasets, linked services, triggers, parameters, and integration runtimes.
• Coordinate orchestration between Airflow and Azure Data Factory while selecting the appropriate platform for different workloads.
• Develop, optimize, and troubleshoot SQL Server queries, stored procedures, transformations, incremental loads, and data extraction processes.
• Build reusable Airflow operators, hooks, sensors, shared libraries, pipeline templates, and engineering components.
• Implement monitoring, logging, dashboards, operational metrics, and alerting to improve pipeline observability and production support.
• Define and improve retry, restart, backfill, catch-up, failure-recovery, and disaster-recovery procedures.
• Partner with infrastructure and security teams to strengthen secrets management, identity, access controls, certificates, network connectivity, and private endpoints.
• Maintain architecture diagrams, deployment documentation, operational procedures, and production-support runbooks.
• Mentor data engineers on Airflow, Python, CI/CD, automated testing, troubleshooting, and production engineering practices.
• Evaluate emerging technologies such as Databricks and recommend future-state improvements to the enterprise data platform.

Ideal Candidate

• 5+ years of experience in Data Engineering, Data Platform Engineering, DevOps, Software Engineering, or data platform operations.
• Strong production experience with Apache Airflow and hands-on experience with Astronomer or a comparable managed Airflow platform.
• Advanced Python skills with significant experience developing, testing, and troubleshooting Airflow DAGs.
• Strong knowledge of Airflow concepts including task groups, operators, hooks, sensors, callbacks, dynamic task mapping, scheduling, connections, variables, plugins, backfills, and recovery.
• Proven experience designing and supporting CI/CD pipelines for data platforms and orchestration workloads.
• Experience managing deployments across controlled development, QA, UAT, and production environments.
• Strong experience with Azure Data Factory, including pipeline development, triggers, datasets, linked services, parameters, and integration runtimes.
• Advanced Microsoft SQL Server experience, including complex SQL, stored procedures, performance tuning, incremental loads, troubleshooting, and reconciliation.
• Experience integrating databases, REST APIs, cloud storage, file systems, and enterprise applications.
• Strong understanding of automated testing, observability, logging, monitoring, alerting, retry strategies, and production failure recovery.
• Ability to diagnose complex production issues spanning application, orchestration, database, infrastructure, and network layers.
• Strong understanding of secure configuration, secrets management, identity, authentication, and access control.
• Experience working collaboratively with data engineers, architects, infrastructure teams, security teams, DBAs, and application developers.

Preferred Qualifications

• Bachelor’s degree in Computer Science, Data Engineering, Software Engineering, Information Systems, or a related discipline, or equivalent professional experience.
• Experience with CI/CD platforms such as Azure DevOps Pipelines, GitHub Actions, GitLab CI, or Jenkins.
• Familiarity with Docker, Kubernetes, Airflow on Kubernetes, and Terraform.
• Experience with Azure Key Vault, managed identities, service principals, and ADLS Gen2.
• Familiarity with Databricks, Spark, PySpark, Delta Lake, Kafka, and dbt.
• Experience with monitoring and observability technologies such as OpenTelemetry, Azure Monitor, Grafana, or Prometheus.
• Knowledge of Data Vault 2.0, VaultSpeed, or modern enterprise data architecture patterns.
• Experience supporting manufacturing, ERP, finance, supply-chain, or operational data environments.

Professional Attributes

• Highly analytical with strong troubleshooting and root-cause analysis skills.
• Proactive and focused on preventing recurring production issues rather than relying on reactive support.
• Strong sense of ownership and accountability for platform reliability and engineering quality.
• Process-oriented with a passion for automation, standardization, and continuous improvement.
• Comfortable mentoring engineers and establishing technical best practices across teams.
• Collaborative communicator who can work effectively across engineering, infrastructure, security, architecture, and business teams.
• Detail-oriented and capable of maintaining high engineering standards across complex enterprise environments.

Benefits and Highlights

• Contract-to-hire opportunity with the potential to transition into a full-time position following the contract period.
• Competitive W2 compensation ranging from $70–$75 per hour.
• Medical, dental, and vision benefits available to eligible W2 consultants.
• 401(k) plan with company matching and life insurance benefits.
• Opportunity to take significant ownership of enterprise data platform engineering standards and practices.
• Hands-on exposure to Apache Airflow, Astronomer, Azure Data Factory, SQL Server, automation, and modern cloud data technologies.
• Opportunity to influence future data architecture while mentoring engineers and improving platform reliability across the organization.