AI’s biggest obstacle? Data reliability. Astronomer’s new platform tackles the challenge

Astronomer, the company behind Apache Airflow orchestration software, has launched Astro Observe, marking its expansion from a single-product company into the competitive data operations platform market. The move comes as enterprises struggle to operationalize their AI initiatives and maintain reliable data pipelines at scale.

The new platform aims to help organizations monitor and troubleshoot their data workflows more effectively by combining orchestration and observability capabilities in a single solution. This consolidation could significantly reduce the complexity that many companies face when managing their data infrastructure.

“Previously, our customers would have to come to us for orchestration data pipelines, and they’d have to go figure out a different data observability and Airflow observability vendor,” Julian LaNeve, CTO of Astronomer, said in an interview with VentureBeat. “We’re trying to make that a lot easier for our customers and give them everything in one platform.”

AI-powered predictive analytics aims to prevent pipeline failures

A key differentiator of Astro Observe is its ability to predict potential pipeline failures before they impact business operations. The platform includes an AI-powered “insights engine” that analyzes patterns across hundreds of customer deployments to provide proactive recommendations for optimization.

“We will actually tell people two hours before the SLA is going to happen that they’re likely to miss it because there was some delay far upstream,” LaNeve explained. “That moves people from this very reactive world to a lot more proactive [approach], where you can start to address issues before downstream stakeholders find out.”

The timing is particularly significant as organizations grapple with operationalizing AI models. While much attention has focused on model development, the challenge of maintaining reliable data pipelines to feed these models has become increasingly critical.

“Ultimately, to take these AI use cases from prototype to production, it becomes a data engineering problem at the end of the day,” LaNeve noted. “How do you effectively feed these LLMs the right data on time every time? That’s what data engineers have been doing for many years now.”

Astronomer moves from open source success to enterprise data management

The platform builds on Astronomer’s deep expertise with Apache Airflow, an open-source workflow management platform downloaded more than 30 million times monthly. This represents a significant increase from just four years ago when Airflow 2.0 saw less than a million downloads.

One notable feature is the “global supply chain graph,” which provides visibility into both data lineage and operational dependencies. This helps teams understand complex relationships between different data assets and workflows — which is crucial for maintaining reliability in large-scale deployments.

The platform also introduces a “data product” concept, allowing teams to group related data assets and assign service level agreements (SLAs). This approach helps bridge the gap between technical teams and business stakeholders by providing clear metrics around data reliability and delivery.

Early adopter GumGum, a contextual intelligence company, has already seen benefits from the platform. “Adding data observability alongside orchestration allows us to get ahead of issues before they impact users and downstream systems,” said Brendan Frick, GumGum senior engineering manager at GumGum.

Astronomer’s expansion comes at a time when enterprises are increasingly looking to consolidate their data tooling. With organizations typically juggling eight or more tools from different vendors, the move toward unified platforms could signal a broader shift in the enterprise data management landscape.

The challenge for Astronomer will be competing with established observability players while maintaining its leadership in the orchestration space. However, its deep integration with Airflow and focus on proactive management could give it an edge in the rapidly evolving market for AI infrastructure tools.

Daily insights on business use cases with VB Daily

If you want to impress your boss, VB Daily has you covered. We give you the inside scoop on what companies are doing with generative AI, from regulatory shifts to practical deployments, so you can share insights for maximum ROI.

Read our Privacy Policy

Thanks for subscribing. Check out more VB newsletters here.

An error occured.