The Hortonworks Model

This is a unique moment in time. Fueled by open source, Apache Hadoop has become an essential part of the modern enterprise data architecture and the Hadoop market is accelerating at an amazing rate.

The impressive thing about successful open source projects is the pace of the “release early, release often” development cycle, also known as upstream innovation. The process moves through major and minor releases at a regular clip and the downstream users get to pick the releases and versions they want to consume for their specific needs.

In the case of Apache Hadoop platforms like the Hortonworks Data Platform (HDP), we see the consumption of dozens of specific versions of Apache Software Foundation (ASF) projects:

Assembling a complete platform like HDP requires choosing the right stable version of Apache Hadoop as the foundation and then integrating, and packaging the optimal versions of all the other ASF components into a well-tested, certified data platform.

“Open Source Communities: The Developer Kingdom” adds a little more color around this process.

Working within the community for the enterprise

Since we are committed to delivering HDP completely in the open, we introduce enterprise feature requirements into the public domain and we code to address those requirements. We code and we contribute everything back to the wide array of ASF projects in HDP. Why? Because we want the Hadoop market to work at scale, and in order to make this happen, we know code is king, so we practice what we preach. We contribute everything.

Specifically, we:

1. Innovate
   Innovate within existing ASF projects to accelerate enterprise-focused innovation. Our work on Apache Hadoop YARN, HDFS, and the Stinger Initiative/Hive are great examples.
2. Incubate
   Identify and create new ASF projects that address security, management, operations, and other enterprise needs; Apache Ambari, Apache Falcon, Apache Ranger, Apache Knox, Apache Tez, and Apache Slider are great examples.
3. Acquire and Contribute
   Acquire innovative companies and contribute the IP to the ASF as an Apache incubator project. We acquired XA Secure in 2014 and flipped this commercial software for comprehensive security into open source as Apache Ranger.
4. Partner and Deliver
   Establish joint engineering relationships to accelerate Enterprise Hadoop innovation. Our deep joint engineering work with Microsoft, HP, SAS, Pivotal, Red Hat, Teradata and others are great examples. Microsoft’s recent launch of its Azure HDInsight service on Linux is a great example of what comes from joint engineering.
5. Rally the Ecosystem
   Found enterprise-focused initiatives that rally end users and vendors towards common goals. The Stinger initiative, the Data Governance initiative and the Open Data Platform initiative are great examples.

The enterprise-focused initiatives are an important element of our approach. The Stinger Initiative has successfully rallied contributions from hundreds of developers across dozens of companies in order to address SQL in Hadoop needs for the enterprise. The Data Governance Initiative was recently formed with Aetna, Merck, Target and SAS to address the data stewardship, lineage, lifecycle management, and privacy issues that are increasingly important.

Open Data Platform Initiative (ODP)

The Open Data Platform initiative (ODP), announced this week, aims to rally enterprise end users and vendors alike around a well-defined common core platform (the ODP Core) against which big data solutions can be qualified.

How does the ODP relate to the ASF?

It’s simple. All upstream production happens within the ASF projects according to the ASF governance model. Individuals working for ODP member companies are encouraged to participate and contribute to ASF projects as they see fit and in accordance to ASF processes. Since Hortonworks engineers do all of their coding in these ASF projects, we’re more than happy to help newcomers learn the Apache way and contribute.

The ODP, on the other hand, is focused on enabling downstream consumption of a common set of Hadoop-related components, and more importantly, specific versions of those components. Harmonizing the broader market around Apache Hadoop version 2.6 and Apache Ambari 2.0, for example, will help simplify the onboarding of manageable, YARN-based solutions that can ride atop the common core platform.

Increasing the compatibility among Hadoop-based platforms and solutions will free up the broader big data ecosystem to focus on more important things such as data-driven applications that deliver proactive insights for the business. Innovation will advance even faster in the market with all the ODP members building upon the same downstream Apache Hadoop kernel- Apache Hadoop, Yarn and Apache Ambari.

Modern platform standards are defined by open communities

At Hortonworks, our founding belief is that innovation and adoption of platform technologies like Hadoop is best accomplished through collaborative open source development under the governance model of an entity like the Apache Software Foundation (ASF).

In order to enable a data platform like Hadoop to be easy to use and enterprise-grade, you don’t go it alone. You do it by working with your customers and the broader ecosystem to enable:

• data architects to deeply integrate existing systems with Hadoop;
• developers, data workers, and analysts to build applications quickly and easily; and
• operators and security administrators to deploy, manage, secure, and govern the platform and the applications deployed on it in a consistent way.

Our approach to the market is about enabling our customers to embrace Hadoop in a way that makes sense for their business. It’s about enabling our partners in a way that drives joint value from the alliance in a way that’s respectful to each other in the process. And it’s about rallying a community in a way that drives innovation around shared goals.

Done right, open source promotes an equitable balance of power; done together, it offers a fair exchange of value between vendor and consumer.