Hadoop Summit 2014: Catching up with the latest innovations in the Hadoop Distributed File System

Technologists from around the globe, from vendors, to service providers, to consultants, to customers descended on the San Jose McEnery Convention Centre for the Hadoop Summit to learn from industry peers and leaders about the latest innovations on the Hadoop Distributed File System. The event was also an opportunity for the leading technology houses and startups to exhibit their solutions on the revolutionary platform designed to “tame” Big Data.

Hortonworks President Herb Cunitz opened proceeding with a recap of the progress of the event since its inception in 2008 and how Hadoop has evolved over the years. The conference has grown from an attendance of 200 people to 3200 at this year’s event. There were 126 general sessions – up 22% from last year, and 88 total sponsors – up 30% from last year. The numbers indicate the growing interest in the Hadoop platform within the technology sector, and is amplified by the number of technology firms that have taken notice of the growing influence of Hadoop.

From its humble beginnings as a massively scalable distributed file system to handle single purpose, large volume batch transactions, the platform became a vehicle for early developers to innovate further. The community began to wonder and explore ways to extend the capabilities of Hadoop. When Hadoop became enterprise grade, companies like Cloudera, Hortonworks and MapR were born to apply governance and compliance, security and operationalize the platform.

Innovation on Hadoop has given rise to key access patterns such as interactive queries through Apache Hive, streaming queries through Apache Storm, and in-memory queries such as Apache Spark. Last year’s event also saw the advent of YARN, the data operating platform that enabled Hadoop to become a core data platform.

Hadoop has now emerged as the default platform to manage Big Data and this has not been lost on technology vendors who could not ignore its growing influence in a rapidly changing industry. Many of the technology houses have either partnered with the enterprise Hadoop management companies, built their own Hadoop connectors or started to develop applications that run natively on Hadoop.

What is even more astounding is the number of startups that have emerged to challenge established vendors with niche solutions in what is seen as a return to best of breed applications and solutions. What is certain is that this is only the beginning. A whole new industry has grown around the Hadoop platform and shows no signs of slowing down.

Click on the link to read up on the entire proceedings of the Hadoop Summit Insight.