Managing the lifecycle of big data

Social Media Strategist, Information & Analytics Group

You might not think of data as a living thing, but it does have a life cycle. For some data, it's fleeting, but other data may live for decades. How do you determine how long to keep data and how to handle it while it's in your care? 

In this #BigDataMgmt Twitter chat, we will discuss methods for determining what to store, how and for how long. We will also examine issues around data security and corporate risk mitigation.

Our special guests for the chat are Mike Ferguson (@mikeferguson1), independent analyst and consultant; Mike Martin (@BTRG_MikeMartin), Information Governance Practice Director at BTRG and IBM Champion; and James Kobielus (@jameskobielus), IBM big data evangelist, speaker and writer. Twitter handle @IBMbigdata will be moderating the chat.

You can follow along and join the discussion using the hashtag #BigDataMgmt. Here are the questions we’ll be discussing, as well as reference articles to help inspire the conversation on Wednesday, April 16, 12:00 noon EST.

#BigDataMgmt chat questions

  1. What is the typical life cycle of big data? Is there even such a thing?

  2. Some people say “Store everything forever. You don’t know when you might need it.” Is this wise? Why or why not?

  3. What are recommendations for determining which data to store and for how long?

  4. Is there a best practice of where data should go? Data warehouse, Hadoop, ODS or other?

  5. Are people moving data out of Hadoop and, if so, how?

  6. How can organizations be more agile with test data so they can roll out new functionality faster and more securely?

  7. How do you mitigate risk, while at the same time allowing for insights to be drawn from big data repositories?

  8. What recommendations do you have for people/orgs who want to start managing the lifecycle of their data?

Related Resources

Featured Guests

Mike Ferguson is Managing Director of Intelligent Business Strategies Limited, also based in the UK. As an analyst and consultant with more than 30 years of experience, Mike works at the board level on business strategy and at detailed technical levels on business analytics, big data, data governance, MDM, enterprise architecture, and event processing. 

Mike Martin is practice director of information governance for BTRG. He solves data quality, data growth, data privacy, data security, test data management and compliance problems for Fortune 1000 customers across many industries.

James Kobielus (@jameskobielus) is IBM Senior Program Director of Product Marketing and IBM's big data evangelist. He is an industry veteran, a popular speaker and social media participant and a thought leader in big data, Hadoop, enterprise data warehousing, advanced analytics, business intelligence, data management and next best action technologies.

What is #BigDataMgmt chat?

#BigDataMgmt chat is a weekly conversation every Wednesday at 12:00 p.m. ET, on Twitter. Each week we discuss a different topic around big data management.

How do you join in?

If you use a Twitter client like Tweetdeck or HootSuite, create a search column for the term ‘#BigDataMgmt’.  Then as participants tweet with the #BigDataMgmt hashtag, those tweets will appear in your column. Or you can follow with Tweetchat – – and it automatically adds the #BigDataMgmt hashtag.

How do you participate?

Just jump right in! Review the discussion questions posted in advance so you can prepare your thoughts and answers. When the question is posed, begin your response with A1: for question 1 and A2: for question 2, etc. This makes it easier to follow the conversation throughout the chat. No answer is wrong! We look forward to seeing you at the #BigDataMgmt water cooler hosted by @IBMbigdata.