{"id":73,"date":"2013-06-05T17:05:19","date_gmt":"2013-06-06T00:05:19","guid":{"rendered":"http:\/\/www.riverlog.com\/wordpress\/?p=73"},"modified":"2013-06-05T17:14:59","modified_gmt":"2013-06-06T00:14:59","slug":"crawling-towards-bigdata-analytics-the-10-things-to-know","status":"publish","type":"post","link":"https:\/\/riverlog.bigleafproductions.com\/wordpress\/2013\/06\/crawling-towards-bigdata-analytics-the-10-things-to-know\/","title":{"rendered":"Crawling towards #BigData analytics, the 10 things to know."},"content":{"rendered":"

What to do when you have a #bigdata project ?<\/p>\n

    \n
  1. Analytics & Analysis has \u00a0been there since many years. Enterprises have been sucking in data from large monolithic systems called data warehouse since many years. \u00a0Once retrieved, the data went through phases of analysis for different purposes that included, sales forecasting, weather predictions, medical analytics and so forth that the term analytics has also gone through a paradiagm shift before it is now being viewed more from predictive analytics standpoint.\u00a0 Time takes before a shift.<\/i> \u00a0So predictive analytics is not just with bigdata. With bigdata a new kind of analytics is evolving.<\/li>\n
  2. Proliferation of bigdata tools and services has begun in the bigdata world. \u00a0Data warehouse as a service<\/strong> was being talked about. But, with data security that is still not embracing \u201cthe openness\u201d for obvious reasons, we have not yet seen success except from \u00a0giants such as Amazon, google, emc, Rackspace etc.\u00a0 When talked about bigdata, there is a secondary term that needs to be queried along with the term bigdata and that is existing analytical data<\/strong>. Please review point number 1 on predictive analytics.<\/span><\/li>\n
  3. Various tools since the inception of Hadoop provide some real cool value adds. Pig, Hive, Google\u2019s NoSql, Cassandra and MongoDb, sqoop, Gora, HBase, Avro, combined with machine learning systems provide the pass-throughs to connect, search, filter, retrieve and manipulate data for further analytics.\u00a0 Check out for oracle big data offering from big vendors.<\/a><\/li>\n
  4. Some have tried to explain differences with business intelligence<\/strong> and replaced it with bigdata. While this attempt is good, they are both complimentary. Many fail to realize that there is already huge work done on analytics and so, bigdata work need not necessarily be a heavy weight lifting.<\/li>\n
  5. While choosing tools, existing infrastructure must be studied. The study must focus on reports attained from business intelligence. This can provide insights into data patterns.<\/strong><\/li>\n
  6. Reports are on existing data. But reports provide insights. Reports on certain intervals or peak periods<\/b> of data or on certain intervals can be detected on running complex equation based queries. \u00a0Used these perceptions for trials. Perceptions here can be reality.<\/b><\/li>\n
  7. The first challenge today, irrespective of various tools within the enterprise and that is existing within the open-source world is to choose the tools. While this is less stressful, once after chosen, the deployment and infrastructure setup is the first major challenge today because, getting connection to the valuable data, and getting massive amount this time, becomes real challenging. \u00a0Here is where the data warehouse as a service, enterprises will lean more towards giants.\u00a0 Please see point number 2.<\/span> <\/b><\/li>\n
  8. The crucial part of bigdata analytics involve creating test data during development. When creating test data, try NOT to replicate or avoid depletion of data points<\/strong>. This can create blocks in moving ahead especially when using PFP\/mahout ; for further analysis of data. Who knows what can help.<\/i><\/li>\n
  9. Furthermore, the test system must be thought of beyond a single machine. Most often for parallel processing, multiple nodes are involved. Especially when processing terabytes of data. Either case, when uploading from data repositories, multiple systems get involved anyways and therefore bigdata.<\/li>\n
  10. Before choosing the tools, understand about the tool with respect to bigdata. There are many tools out there which but otherwise does something else. With the bigdata buzzword in action, many vendors incorporate capabilities for bigdata which is good. But that tools might not be the one you are looking for.\u00a0 Once chosen, do a proof of concept. Engage engineers in rubbing shoulder to shoulder with bigdata people. Always good to collaborate. This is not a one person\u2019s meal? Right?\n

    Thoughts beyond usual thinking.<\/strong>
    \nMany are the things that man seeing must understand. Not seeing, how shall he know what lies in the hand of time to come?<\/em> \u2013 Sophocles<\/li>\n<\/ol>\n","protected":false},"excerpt":{"rendered":"

    What to do when you have a #bigdata project ? Analytics & Analysis has \u00a0been there since many years. Enterprises … <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[1],"tags":[],"yoast_head":"\nCrawling towards #BigData analytics, the 10 things to know. - AI, Generative AI, Cybersecurity, and Cloud: The Power Quartet Driving Enterprise Software Evolution<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/riverlog.bigleafproductions.com\/wordpress\/2013\/06\/crawling-towards-bigdata-analytics-the-10-things-to-know\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Crawling towards #BigData analytics, the 10 things to know. - AI, Generative AI, Cybersecurity, and Cloud: The Power Quartet Driving Enterprise Software Evolution\" \/>\n<meta property=\"og:description\" content=\"What to do when you have a #bigdata project ? Analytics & Analysis has \u00a0been there since many years. Enterprises …\" \/>\n<meta property=\"og:url\" content=\"https:\/\/riverlog.bigleafproductions.com\/wordpress\/2013\/06\/crawling-towards-bigdata-analytics-the-10-things-to-know\/\" \/>\n<meta property=\"og:site_name\" content=\"AI, Generative AI, Cybersecurity, and Cloud: The Power Quartet Driving Enterprise Software Evolution\" \/>\n<meta property=\"article:publisher\" content=\"http:\/\/www.facebook.com\/riverlog\" \/>\n<meta property=\"article:published_time\" content=\"2013-06-06T00:05:19+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2013-06-06T00:14:59+00:00\" \/>\n<meta name=\"author\" content=\"Sunny Menon\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@sunnymenon\" \/>\n<meta name=\"twitter:site\" content=\"@sunnymenon\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Sunny Menon\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/riverlog.bigleafproductions.com\/wordpress\/2013\/06\/crawling-towards-bigdata-analytics-the-10-things-to-know\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/riverlog.bigleafproductions.com\/wordpress\/2013\/06\/crawling-towards-bigdata-analytics-the-10-things-to-know\/\"},\"author\":{\"name\":\"Sunny Menon\",\"@id\":\"https:\/\/riverlog.bigleafproductions.com\/wordpress\/#\/schema\/person\/eb991291ee063bd60d69090852e2f5a0\"},\"headline\":\"Crawling towards #BigData analytics, the 10 things to know.\",\"datePublished\":\"2013-06-06T00:05:19+00:00\",\"dateModified\":\"2013-06-06T00:14:59+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/riverlog.bigleafproductions.com\/wordpress\/2013\/06\/crawling-towards-bigdata-analytics-the-10-things-to-know\/\"},\"wordCount\":638,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/riverlog.bigleafproductions.com\/wordpress\/#organization\"},\"articleSection\":[\"Big Data BigData Cloud\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/riverlog.bigleafproductions.com\/wordpress\/2013\/06\/crawling-towards-bigdata-analytics-the-10-things-to-know\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/riverlog.bigleafproductions.com\/wordpress\/2013\/06\/crawling-towards-bigdata-analytics-the-10-things-to-know\/\",\"url\":\"https:\/\/riverlog.bigleafproductions.com\/wordpress\/2013\/06\/crawling-towards-bigdata-analytics-the-10-things-to-know\/\",\"name\":\"Crawling towards #BigData analytics, the 10 things to know. - AI, Generative AI, Cybersecurity, and Cloud: The Power Quartet Driving Enterprise Software Evolution\",\"isPartOf\":{\"@id\":\"https:\/\/riverlog.bigleafproductions.com\/wordpress\/#website\"},\"datePublished\":\"2013-06-06T00:05:19+00:00\",\"dateModified\":\"2013-06-06T00:14:59+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/riverlog.bigleafproductions.com\/wordpress\/2013\/06\/crawling-towards-bigdata-analytics-the-10-things-to-know\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/riverlog.bigleafproductions.com\/wordpress\/2013\/06\/crawling-towards-bigdata-analytics-the-10-things-to-know\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/riverlog.bigleafproductions.com\/wordpress\/2013\/06\/crawling-towards-bigdata-analytics-the-10-things-to-know\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/riverlog.bigleafproductions.com\/wordpress\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Crawling towards #BigData analytics, the 10 things to know.\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/riverlog.bigleafproductions.com\/wordpress\/#website\",\"url\":\"https:\/\/riverlog.bigleafproductions.com\/wordpress\/\",\"name\":\"The BigData,Artificial Intelligence, Enterprise Process Modernization, Startups and other day to day thinking.\",\"description\":\"Weblog By RiverLog\",\"publisher\":{\"@id\":\"https:\/\/riverlog.bigleafproductions.com\/wordpress\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/riverlog.bigleafproductions.com\/wordpress\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/riverlog.bigleafproductions.com\/wordpress\/#organization\",\"name\":\"RiverLog Software Consulting & Advisory Services\",\"url\":\"https:\/\/riverlog.bigleafproductions.com\/wordpress\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/riverlog.bigleafproductions.com\/wordpress\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/riverlog.bigleafproductions.com\/wordpress\/wp-content\/riverlog_logo_Enhanced_square.png\",\"contentUrl\":\"https:\/\/riverlog.bigleafproductions.com\/wordpress\/wp-content\/riverlog_logo_Enhanced_square.png\",\"width\":195,\"height\":195,\"caption\":\"RiverLog Software Consulting & Advisory Services\"},\"image\":{\"@id\":\"https:\/\/riverlog.bigleafproductions.com\/wordpress\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"http:\/\/www.facebook.com\/riverlog\",\"https:\/\/twitter.com\/sunnymenon\",\"https:\/\/www.linkedin.com\/in\/sunnymenon10\/\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/riverlog.bigleafproductions.com\/wordpress\/#\/schema\/person\/eb991291ee063bd60d69090852e2f5a0\",\"name\":\"Sunny Menon\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/riverlog.bigleafproductions.com\/wordpress\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/139eb39436cf90d010303dd411d67d0d?s=96&d=wavatar&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/139eb39436cf90d010303dd411d67d0d?s=96&d=wavatar&r=g\",\"caption\":\"Sunny Menon\"},\"description\":\"Sunny Menon is a software executive with large enterprise architecture background with over 20+ years of experience in the design, architecture, development of high volume enterprise applications. He has experience enabling cloud environment for enterprise applications. Designed and developed a bigdata products. He has helped #startups evolve from conceptual stages through definition of the actual product by aligning them with industry requirements, developing proof-of-concept and demonstrating the product thereby, helping in seeking funding from VCs. He has extensive experience in the integration of large enterprise applications, middle-ware and modernization of enterprise applications centered around SOA\/SaaS\/PaaS\/Cloud environments. At night, he enjoys 'staring' at the night skies and thinks, twinkle twinkle little star, how I STILL WONDER what you are.... He is a cruel poet who walks bare foot at times, to feel the beauty of the earth. Advisory Board Member and Investor : RiverLog Software Technical advisory to SOADevelopers.com Executive Board Member - Stencil Research Strategist : TechBeat Conference.com CISO - Paragon Security\",\"sameAs\":[\"http:\/\/riverlog.com\"],\"url\":\"https:\/\/riverlog.bigleafproductions.com\/wordpress\/author\/admin\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Crawling towards #BigData analytics, the 10 things to know. - AI, Generative AI, Cybersecurity, and Cloud: The Power Quartet Driving Enterprise Software Evolution","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/riverlog.bigleafproductions.com\/wordpress\/2013\/06\/crawling-towards-bigdata-analytics-the-10-things-to-know\/","og_locale":"en_US","og_type":"article","og_title":"Crawling towards #BigData analytics, the 10 things to know. - AI, Generative AI, Cybersecurity, and Cloud: The Power Quartet Driving Enterprise Software Evolution","og_description":"What to do when you have a #bigdata project ? Analytics & Analysis has \u00a0been there since many years. Enterprises …","og_url":"https:\/\/riverlog.bigleafproductions.com\/wordpress\/2013\/06\/crawling-towards-bigdata-analytics-the-10-things-to-know\/","og_site_name":"AI, Generative AI, Cybersecurity, and Cloud: The Power Quartet Driving Enterprise Software Evolution","article_publisher":"http:\/\/www.facebook.com\/riverlog","article_published_time":"2013-06-06T00:05:19+00:00","article_modified_time":"2013-06-06T00:14:59+00:00","author":"Sunny Menon","twitter_card":"summary_large_image","twitter_creator":"@sunnymenon","twitter_site":"@sunnymenon","twitter_misc":{"Written by":"Sunny Menon","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/riverlog.bigleafproductions.com\/wordpress\/2013\/06\/crawling-towards-bigdata-analytics-the-10-things-to-know\/#article","isPartOf":{"@id":"https:\/\/riverlog.bigleafproductions.com\/wordpress\/2013\/06\/crawling-towards-bigdata-analytics-the-10-things-to-know\/"},"author":{"name":"Sunny Menon","@id":"https:\/\/riverlog.bigleafproductions.com\/wordpress\/#\/schema\/person\/eb991291ee063bd60d69090852e2f5a0"},"headline":"Crawling towards #BigData analytics, the 10 things to know.","datePublished":"2013-06-06T00:05:19+00:00","dateModified":"2013-06-06T00:14:59+00:00","mainEntityOfPage":{"@id":"https:\/\/riverlog.bigleafproductions.com\/wordpress\/2013\/06\/crawling-towards-bigdata-analytics-the-10-things-to-know\/"},"wordCount":638,"commentCount":0,"publisher":{"@id":"https:\/\/riverlog.bigleafproductions.com\/wordpress\/#organization"},"articleSection":["Big Data BigData Cloud"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/riverlog.bigleafproductions.com\/wordpress\/2013\/06\/crawling-towards-bigdata-analytics-the-10-things-to-know\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/riverlog.bigleafproductions.com\/wordpress\/2013\/06\/crawling-towards-bigdata-analytics-the-10-things-to-know\/","url":"https:\/\/riverlog.bigleafproductions.com\/wordpress\/2013\/06\/crawling-towards-bigdata-analytics-the-10-things-to-know\/","name":"Crawling towards #BigData analytics, the 10 things to know. - AI, Generative AI, Cybersecurity, and Cloud: The Power Quartet Driving Enterprise Software Evolution","isPartOf":{"@id":"https:\/\/riverlog.bigleafproductions.com\/wordpress\/#website"},"datePublished":"2013-06-06T00:05:19+00:00","dateModified":"2013-06-06T00:14:59+00:00","breadcrumb":{"@id":"https:\/\/riverlog.bigleafproductions.com\/wordpress\/2013\/06\/crawling-towards-bigdata-analytics-the-10-things-to-know\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/riverlog.bigleafproductions.com\/wordpress\/2013\/06\/crawling-towards-bigdata-analytics-the-10-things-to-know\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/riverlog.bigleafproductions.com\/wordpress\/2013\/06\/crawling-towards-bigdata-analytics-the-10-things-to-know\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/riverlog.bigleafproductions.com\/wordpress\/"},{"@type":"ListItem","position":2,"name":"Crawling towards #BigData analytics, the 10 things to know."}]},{"@type":"WebSite","@id":"https:\/\/riverlog.bigleafproductions.com\/wordpress\/#website","url":"https:\/\/riverlog.bigleafproductions.com\/wordpress\/","name":"The BigData,Artificial Intelligence, Enterprise Process Modernization, Startups and other day to day thinking.","description":"Weblog By RiverLog","publisher":{"@id":"https:\/\/riverlog.bigleafproductions.com\/wordpress\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/riverlog.bigleafproductions.com\/wordpress\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/riverlog.bigleafproductions.com\/wordpress\/#organization","name":"RiverLog Software Consulting & Advisory Services","url":"https:\/\/riverlog.bigleafproductions.com\/wordpress\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/riverlog.bigleafproductions.com\/wordpress\/#\/schema\/logo\/image\/","url":"https:\/\/riverlog.bigleafproductions.com\/wordpress\/wp-content\/riverlog_logo_Enhanced_square.png","contentUrl":"https:\/\/riverlog.bigleafproductions.com\/wordpress\/wp-content\/riverlog_logo_Enhanced_square.png","width":195,"height":195,"caption":"RiverLog Software Consulting & Advisory Services"},"image":{"@id":"https:\/\/riverlog.bigleafproductions.com\/wordpress\/#\/schema\/logo\/image\/"},"sameAs":["http:\/\/www.facebook.com\/riverlog","https:\/\/twitter.com\/sunnymenon","https:\/\/www.linkedin.com\/in\/sunnymenon10\/"]},{"@type":"Person","@id":"https:\/\/riverlog.bigleafproductions.com\/wordpress\/#\/schema\/person\/eb991291ee063bd60d69090852e2f5a0","name":"Sunny Menon","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/riverlog.bigleafproductions.com\/wordpress\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/139eb39436cf90d010303dd411d67d0d?s=96&d=wavatar&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/139eb39436cf90d010303dd411d67d0d?s=96&d=wavatar&r=g","caption":"Sunny Menon"},"description":"Sunny Menon is a software executive with large enterprise architecture background with over 20+ years of experience in the design, architecture, development of high volume enterprise applications. He has experience enabling cloud environment for enterprise applications. Designed and developed a bigdata products. He has helped #startups evolve from conceptual stages through definition of the actual product by aligning them with industry requirements, developing proof-of-concept and demonstrating the product thereby, helping in seeking funding from VCs. He has extensive experience in the integration of large enterprise applications, middle-ware and modernization of enterprise applications centered around SOA\/SaaS\/PaaS\/Cloud environments. At night, he enjoys 'staring' at the night skies and thinks, twinkle twinkle little star, how I STILL WONDER what you are.... He is a cruel poet who walks bare foot at times, to feel the beauty of the earth. Advisory Board Member and Investor : RiverLog Software Technical advisory to SOADevelopers.com Executive Board Member - Stencil Research Strategist : TechBeat Conference.com CISO - Paragon Security","sameAs":["http:\/\/riverlog.com"],"url":"https:\/\/riverlog.bigleafproductions.com\/wordpress\/author\/admin\/"}]}},"_links":{"self":[{"href":"https:\/\/riverlog.bigleafproductions.com\/wordpress\/wp-json\/wp\/v2\/posts\/73"}],"collection":[{"href":"https:\/\/riverlog.bigleafproductions.com\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/riverlog.bigleafproductions.com\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/riverlog.bigleafproductions.com\/wordpress\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/riverlog.bigleafproductions.com\/wordpress\/wp-json\/wp\/v2\/comments?post=73"}],"version-history":[{"count":7,"href":"https:\/\/riverlog.bigleafproductions.com\/wordpress\/wp-json\/wp\/v2\/posts\/73\/revisions"}],"predecessor-version":[{"id":76,"href":"https:\/\/riverlog.bigleafproductions.com\/wordpress\/wp-json\/wp\/v2\/posts\/73\/revisions\/76"}],"wp:attachment":[{"href":"https:\/\/riverlog.bigleafproductions.com\/wordpress\/wp-json\/wp\/v2\/media?parent=73"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/riverlog.bigleafproductions.com\/wordpress\/wp-json\/wp\/v2\/categories?post=73"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/riverlog.bigleafproductions.com\/wordpress\/wp-json\/wp\/v2\/tags?post=73"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}