Table of Contents
show
How was 2014 for Big Data, Analytics and Crawling?
In 2014, Big data, analytics and data crawling were at once hailed as “the future” of industry and admonished as a “bubble waiting to explode”. 2014 was the first year when enterprises became serious about big data. Enterprises asked questions on how to begin with Big Data, how to manage data distribution, and more importantly, how to get data! Twelve months hence, perspectives on Big Data, analytics and business intelligence have changed.
Changing Perspectives
Early in the year, the call was loud about how data wasn’t big if it didn’t have big impact. But that’s naïve to not see the sheer volume of data being generated and used. It is, quite literally, big!
How do we know this? As a Data-as-a-Service provider, we catered to clients who had scraping and crawling requirements worth millions of data points. We saw an exponential increase in e-commerce data, pricing data, and (not surprisingly) enterprise-valued data that added to business intelligence use-cases. Data crawling and scraping providers were busy trying to meet demands of the burgeoning data market.
What changed radically was how data was viewed. While many enterprises vowed to have significantly sized datasets, they still clamoured for correlation between these datasets. Data visualization and data scientists emerged as the champions of a data-driven way of doing business.
Identifying what data would work best for a client and how it would be relevant to them was important to us at PromptCloud. As a DaaS provider, what good is scraping websites and data crawling if it fails to align to use-case and doesn’t align with results desired?
Open Source was the Key
2014 promised more on Hadoop 2.0 and raised the bar dramatically. The consensus that emerged was that Hadoop gave a “more rounded” perspective to the sheer volume of data generated and collected.
More applications were created to derive value from new data types. Newer sources like sensors/machines, server logs, clickstreams, and social media formed a lake with Hadoop bridging the divide between deep insight and disparate data sets. The analogy is simple, with Hadoop enterprises can fish in an ocean yet still bait exactly the catch they want! This change was brought about by relational databases, which are more scalable and less limiting than traditional technologies. When used in conjunction with HBase, Cassandra and MongoDB, Hadoop became cornerstone to build technology-driven business on.
Yet, despite its ability to linearly scale data and storage and data processing ability, Hadoop alternatives, like “in-memory databases” are now vying for attention of data crunchers.Interestingly, Hortonworks, the Yahoo Big data-spin off and open source software project Apache Hadoop developer, went public in 2014!
See our popular deck on data & Finance here: 10 Big Data Trends in Finance
Finance became Data-Driven
2014 saw significant financial institutions shifting to a more data-driven policy. This year, banking organizations realized that their biggest strength was not capital, but data. In the new digital economy model, data is something that needs to be capitalized, and many successfully adopted customized IT systems to harness this data.
Results of this shift can be seen in cost-reduction in fraud management; while account management is enhanced with new customer insights gleaned from data. More than anything, Finance saw improved relationship management using analytics and Big Data. By collecting data like share price movements, credit lines data and other internal data, understanding of key client businesses became better. E-transactions, PayPal and other systems, provide another avenue from which financial data can be collected and used for business intelligence.
The potential of the payments business is to maximise access to valuable customer data. With e-commerce booming, financial transactions saw a steep increase as well.
Consumer Data was Most-Wanted
We also found out this year that social media data was most popular. It was important for brands / marketers to understand how consumer sentiment moved. Twitter crawling, our niche feature, emerged as a popular feature. Here we extracted data (tweets!) based on specific keywords and even geographical locations, as per client’s request. Where Twitter data was inadequate, we saw demand for mass-scale crawls that enabled us to get reviews and comments from blogs and forums which was processed for consumer sentiment analysis.
E-commerce leveraged customer data for gratification, which emerged as a significant derivative of having consumed data and predicting outcomes based on analytics. E-commerce became more personalized in 2014, all thanks to big data analytics. By personalization, marketers understood how to categorize customers from their databases and also how to effectively use different “content on the screen” to specifically display targeted messages to users.
Another aspect where data flourished vis-a-vis the consumer was the development of mobile apps and wearable devices. The more gadgets that collect data flourish, the more they need data! The Internet of Things and its development hinges on quality data being collected and analyzed. Unsurprisingly, the demand for real-time data access topped the charts. Real-time data access helps gauge not just response but also gratification.
Read more about Internet of Things here
FOR THE FUTURE
The insistence on seeing credible data was also high in 2014, perhaps an indication for future that as a DaaS provider, the value-addition was not just data aggregation but also structuring data so to make it analytics-ready.
From our perspective the industry needs to re-evaluate the Holy Grail of Big Data. Getting back to the basics is important. What we believe, is that as long as data quality is not addressed, neither investment nor deployment of data analytics is going to be of any help. Here’s wishing for a data-driven 2015!
Image Credits: mycustomer
