Enterprises are taking every step to tap into the realms of bigdata. How much is bigdata has already been debated. It seems, there are valuable and futuristic information hidden within big data clusters, no doubt – Old jungle saying. If so, then why is that, this valuable information is unable to be retrieved. Speed is not a problem anymore. Even if speed is a problem, people may be ready to wait a while, to know “The Future“. Marketing cut throats, business greedy owners and others who have nothing to loose and everything to earn, have shifted the needs to something else and focus to elsewhere in space. Technological erosion is due to this fact, wherein personal needs and vested interests led people, in doing deeds and buying things, which people don’t need it in the first place. A man buys the material he never ever wanted and would never be needing it at all, in his life time. All done through the influence of the powerful media. Nevertheless,the big data implementations currently exist are nothing but retrieval of traditional reports. Projects ripple across CIO’s but finally find its resting place within IT and becomes an IT project. Lost in translation, the CIO is now provided with traditional reports. Here are top ten reasons, BigData will take a second jump soon.
1. Requirement for experienced talent pops up. For some time, companies have been focusing on recruiting fresh talent. While this is a great “venture”, it can also cause what can be called as “beginners fall“. Not knowing the complexity of applications, these fresh recruits can go ahead and create solutions that may not solve the actual problem. But this trend seems to be slowly diminishing, especially within the #bigdata world. People seem to be looking for experienced hands. Conferences are filled with long bearded masters in the domain. Requirements are flooded with minimum eight year experience statements.
2.In the footsteps of a giant:So far traditional reports were being provided to the office of CIO and Marketing, business and other stake holders. With more experienced hands working on #bigdata, analytics, look and feel seems fresh. There is still time to mature, but it appears to be in the right path. Giants have already reaped the fruit of #bigdata, because investment was heavy and they have seen the rise of big data, years ahead of many. While they themselves are learning, many lessons have come out.
3.Bigdata should not be an IT project. Infrastructure tools are plenty. But with huge data that requires processing, even with MapReduce, this task was not understood in reality. Distributed systems and coding with that paradigm has always been a challenge. With more experienced technical hands, this challenge is being met with. People have understood the potential of MR’s and other distributed processing systems. Multi-core programming concepts and capabilities, are also becoming helpful.
4. Service enablement API provisions. Data always resided within several sub systems within an enterprise. With more integration between business process and more robust API frameworks available and with distributed processing capabilities, data retrievals, processing and analytics have become more seamless than ever before.
5. Technological challenges: A fair number of tools have already been made available to developers. Although infrastructure tools dominate, applications that focus on domain specific data is being released from time to time. This helps organizations in achieving #bigdataanalytics. There are a lot of confusions here. With big data hype in the market, all kinds of companies are embedding the keyword big data and internet is proliferated with such useless documents that has no relevance to big data. RSS syndication may be of some help with specialized searches. Yet, challenge on accurate information retrieval exists.
6.Data Scientists have entered the market with data mining knowledge and not very much focussed on big data. Although we see challenges in the usage of big data tools by these data scientists, we still have many algorithms, complex data science related concepts evolving that may provide better analytics. There is always a tendency to go back to traditional reports as opposed to analytics. Overcoming this trend need to take place.
7.Data accessibility: For achieving accurate analytics, not only technology alone or complex algorithms or mathematical models and visualization capabilities would not suffice. Most important factor, DATA must be available within hands reach. With extensive compliance needs and disparately residing data and most importantly, unstructured and semi-structured data within enterprises, will make it hard on enterprises to give the data out for processing, even internally.
8.BigData as a service:- Bigdata means analytics and not report. This will be a huge challenge, given the point 7 above. However, this possibility of having to get the job done on a service based approach is a huge benefit because, this reduces tremendous research, design and development time put forth by enterprises and they can simply rely on third party vendors who may have done this job anyway. But with point number 7 in perspectives, this handover or usage of such software by enterprises will be an issue. But because of selling pressures within big data vendors, these saas models will turn into outright traditional licensing models very quickly. This change in licensing model may perhaps help in taking big data analytics to next level.
9.Hidden Data: The secret of business is to know something that nobody else knows ~Aristotle . Given that the data resides in unstructured form, arrival of data within the enterprise through sensory devices , to get to what is not known what resides there requires expertise. Data scientists should be relied upon. Mathematical models and probability thought models may only be a beginning point. Abstraction of probability has to be broken open to derive patterns or predictability. Predictability may not be the only derivation that people should go after.
10.Breaking open the technological challenge: Bigdata projects always spin through different business units. Most often , it arises from the office of CIO. Many evolve from the demands of marketing and sales. May what it be, today, it all ends up within the engineering/IT division. This flow need to stop. BigData should not be an IT project. If it becomes an IT project, what we will get back will be nothing but reports. Tools must be designed and developed by IT but the final attainment of results must be done by the stake holder. Moreover, big data should be an enterprise wide project and not segregated within certain business units. Therefore infrastructure for dealing with big data should be supported and mentored by engineering/IT ONLY and the owners must be the actual INDEPENDENT users. The job must be delegated and executed by the stake holder rather than looking at big data as a IT project.
