Evaluating the experience of using IBM Data Science Experience
IBM Data Science Experience (DSX), which is still in open beta, is a new cloud-based IBM solution for data scientists. Because DSX is a tool for analyzing data and collaborating in data science projects, we decided to—what some refer to in programmer parlance—dog food it. In other words, we used DSX to analyze DSX users’ experience based on our analysis of DSX usage data.
Dog fooding DSX involves collecting the data mainly using two tools: segment and intercom. We use segment to track users’ actions, and we use intercom to engage in conversations with users. The DSX server stores actions and conversations among users in a relational database. From a DSX Jupyter notebook, we access the data to explore, analyze and visualize it:
In our mission to analyze DSX user engagement and adoption, we came up with a way to measure user activity. For each user, we define:
active_days = number of days when the user triggered an action
Approximately 30 different actions are tracked on the DSX server. For each user, we count the number of times the user executes each of the tracked actions. Then we compute linear correlations between active_days and the counts of each of the actions. As a result, we find actions that are statistically significant and strongly correlated with the activity score of each user. For example, the action of creating new Jupyter notebooks is positive and strongly correlated with the active_days count.
To quickly understand the pieces of DSX that are not intuitive or buggy, we listen to users through the intercom conversations. But reading all the conversations one by one is very time-consuming. Therefore, we applied natural-language processing tools through the IBM Watson Alchemy application programming interface (API) to process the conversations and perform keyword extractions with a sentiment analysis attached. As a result of our analysis, the DSX design and development team can be notified with the aspects of the product that are confusing or unreliable because of bugs.
If you are interested in more details on the data science behind this project, register today for IBM Insight at World of Watson 2016, and join us at the presentation, How We Use Data Science Experience to Analyze Data Science Experience, Tuesday, 25 October 2016, at 1 p.m. Pacific in Las Vegas, Nevada.