About IrthSolutions
Irth Solutionsis a leading provider of cloud-based SaaS software for damage prevention, asset integrity, stakeholder engagement and land management, helping energy, utility, telecom, and infrastructure companies protect their critical network infrastructure. With nearly three decades of industry experience, Irth serves customers across North America and continues to expand its platform with new data-driven and AI-powered capabilities.
Data Engineer – Insights (AI/ML)
Location: Remote – India
Department: Insights (AI/ML)
Reports to: Data Platform & Analytics Manager
About the Role
Irth is building a modern, multi-cloud, enterprise-grade data estate—a unified Databricks-based data platform that centralizes data across Irth’s products and cloud environments, including AWS, Azure, and GCP.
As a Data Engineer, you will play a hands-on implementation role, working closely with the Senior Data Architect to bring the enterprise data platform vision to life.
You will design and develop data pipelines based on established architectural patterns, implement data quality and governance controls, build Delta Lake and medallion architecture solutions, and help operationalize the new data platform.
This is an excellent opportunity for a mid-level Data Engineer looking to deepen their expertise in Databricks, Apache Spark, cloud data engineering, and modern lakehouse architecture while working in a multi-cloud enterprise environment.
Key Responsibilities
1. Data Pipeline Development – Primary Responsibility
Build, maintain, and enhance data ingestion pipelines across AWS, Azure, and GCP, following architecture and engineering patterns established by the Senior Data Architect.
Develop both batch and streaming pipelines using:
Databricks Workflows
Apache Spark / PySpark
SQL
Delta LiveTables
Databricks Lakeflowcomponents
Implement Bronze → Silver → Gold medallion architecture patterns for ingestion, transformation, cleansing, and standardization.
Implement Change Data Capture(CDC) and Slowly Changing Dimensions (SCD Type 1 and Type 2).
Handle schema evolution and changing source-system structures.
Implement data validation, reconciliation, and quality rules as part of pipeline processing.
Build reusable and maintainable pipeline components following established engineering standards.
2. Platform & Storage Implementation
Configure and maintain Delta Lake storage structures, tables, schemas, partitions, and optimization routines.
Apply Delta Lakeperformance and maintenance practices, including:
OPTIMIZE
Z-ORDER
VACUUM
Appropriate partitioning and file-management strategies
Assist with implementation of metadata, cataloging, and lineage standards using Unity Catalog.
Support integration between cloud storage platforms and Databricks, including:
Amazon S3 → Databricks
Azure Storage → Databricks
Google CloudStorage → Databricks
Assist with implementation of scalable storage and processing patterns defined by the Data Architect.
3. Data Governance, Quality & Compliance Enablement
Implement automated data-quality checks, profiling, validation, and monitoring in accordance with enterprise governance standards.
Apply data-quality rules at appropriate stages of the Bronze, Silver, and Gold layers.
Implement RBAC policies, security controls, and data-classification tags defined by the enterprise governance model.
Support implementation of metadata and lineage mapping across Unity Catalog and Microsoft Purview.
Help ensure datasets are properly documented, classified, governed, and discoverable.
Support remediation of data-quality and governance issues identified through monitoring or reviews.
4. Orchestration, Automation & Operational Support
Build, schedule, monitor, and maintain production workflows using:
Databricks Workflows
Delta LiveTables
Azure Data Factory(ADF)
Other approved orchestration tools
Contribute to CI/CD pipelines for data-engineering code, including source control, automated testing, deployment, and environment management.
Support DEV → QA → PROD promotion processes.
Monitor production pipelines and respond to failures and data-quality issues.
Troubleshoot failed jobs, investigate root causes, and support pipeline recovery.
Perform performance tuning across Spark jobs, SQL workloads, Delta tables, and data pipelines.
Participate in operational improvements that increase pipeline reliability, scalability, and cost efficiency.
5. Collaboration & Documentation
Work directly with the Senior Data Architect to translate architecture designs and technical standards into actionable implementation tasks.
Participate in architecture reviews, technical design discussions, coding reviews, and engineering standards meetings.
Collaborate with Data Scientists, ML Engineers, Analysts, Product teams, and other engineering stakeholders to understand data requirements.
Document:
Data pipelines
Data flows
Data dictionaries
Transformation logic
Data-quality rules
Test cases
Job schedules
Operational procedures
Maintain clear and accurate technical documentation to support platform adoption, troubleshooting, and future development.
Provide implementation feedback to the Data Architect and identify opportunities to improve platform patterns, tooling, and developer experience.
Role Scope
This is primarily an implementation-focused Data Engineering role. The Senior Data Architect will establish the overall platform architecture, standards, and design patterns; the Data Engineer will translate those patterns into reliable, production-ready pipelines and platform capabilities.
The role provides an opportunity to gain deeper hands-on experience with Databricks, Spark, Delta Lake, Unity Catalog, cloud data platforms, data governance, and multi-cloud lakehouse engineering while contributing to a strategic enterprise data platform.
Requirements
Qualifications
Required Qualifications
3–5 years of experience in Data Engineering, ETL development, or cloud data platform engineering.
Hands-on experience with Databricks, Apache Spark, PySpark, or other distributed data-processing technologies.
Strong proficiency in SQL, including structured data transformation, joins, aggregations, and performance-aware query development.
Experience working with at least one major cloud platform, with Microsoft Azure preferred; AWS and/or GCP experience is also valuable.
Understanding of core data-engineering concepts, including:
Data modeling
Data quality
Schema evolution
Data validation
Pipeline monitoring and troubleshooting
Basic understanding of data-security practices, including:
Role-Based Access Control (RBAC)
Encryption
Credential and secret management
Secure access to cloud and data-platform resources
Preferred Qualifications
Hands-on or working knowledge of Delta Lake, medallion architecture, and modern lakehouse best practices.
Experience with metadata, cataloging, and governance platforms such as:
Unity Catalog
Microsoft Purview
AWS Glue DataCatalog
Similar enterprise metadata and data-governance tools
Experience with workflow orchestration and scheduling technologies, such as:
Azure Data Factory(ADF)
Databricks Workflows
Apache Airflow
Databricks Jobs / DBX
Similar orchestration frameworks
Experience with Git-based development, CI/CD, and DevOps practices.
Knowledge or experience in one or more of the following areas:
Geospatial/GIS data
BI semantic layers, particularly Power BI
Data preparation for AI/ML workloads
Relevant cloud or Databricks certifications, such as:
Databricks Data EngineerAssociate
Microsoft Azure Data Engineer Associate(DP-203)
Equivalent cloud or data-engineering certifications
Nice-to-Have Qualifications
Understanding of asset integrity management concepts, including inspection data, risk scoring, corrosion tracking, defect management, and maintenance data as applied to pipeline or utility operations.
Previous experience working with or integrating oil & gas, utility, infrastructure, or pipeline asset data into enterprise data platforms.
Experience working with:
Pipeline and facility data
GIS/geospatial asset data
Inspection and maintenance records
Asset-risk datasets
Familiarity with regulatory, compliance, and audit-reporting requirements associated with pipeline, utility, or asset-integrity data.
Success Metrics
Success in this role will be measured by the engineer’s ability to reliably implement and operationalize the data-platform patterns established by the Data Architect.
Key measures include:
High-quality implementation of ingestion, transformation, data-quality, and governance patterns defined by the Data Architect.
Reliable and maintainable pipelines supporting consistent Bronze → Silver → Gold data flows.
Strong adherence to cataloging, metadata, lineage, security, and data-governance standards.
Reduction in pipeline failures and production incidents through improved monitoring, testing, troubleshooting, and operational practices.
Continuous improvements in pipeline performance, scalability, reliability, and maintainability.
Clear and complete technical documentation covering pipelines, transformations, data-quality rules, and operational procedures.
Effective collaboration with the Senior Data Architect, engineering teams, product teams, and business stakeholders.
Demonstrated ability to take architecture guidance and translate it into production-ready, scalable data-engineering solutions.
Benefits
Benefits
Competitive Salary – A competitive compensation package based on experience and qualifications.
Medical, Dental, and Vision Insurance – Comprehensive insurance coverage to support you and your family.
401(k) Plan with Company Match.
Generous Paid Time Off(PTO) – Time off to support work-life balance and personal needs.
Company-Paid Holidays – Paid holidays throughout the year.
Flexible WorkOptions – Work-from-home opportunities are available, depending on role and business needs.
On-Call Compensation – Additional pay for eligible on-call shifts.
Originally posted on Himalayas