Uncovering user data (Series)

Sat, 26 Sep 2015 00:00:00 +0000

This is a second blog post of this marketing analytics series on a coupon e-commerce website (ponpare.jp). I will review some of the user statistics and purchasing patterns. In the continuing posts, I will cover more in-depth metrics to understand users and their purchasing behaviors.

In [73]:

import os.path
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
import matplotlib.cm as cm
import pandas as pd
%matplotlib inline

user_list = pd.read_csv("user_list.csv")
user_list['REG_DATE'] = pd.to_datetime(user_list['REG_DATE'])
user_list['REG_MONTH'] = user_list['REG_DATE'].dt.month
user_list['REG_HOUR'] = user_list['REG_DATE'].dt.hour

User statistics¶

There are a total of 22,873 unique users in this user list with over 95% active users. Percentages of males and female users are similar with slightly more male users. Looking at active and inactive users separately, while genders represented among active users are similar to that of the total list, there are 15% more males among inactive users who have withdrawn their registration.

In [52]:

total_users = user_list['USER_ID_hash'].unique().shape[0]
print '*Total # of users: ', total_users
print '\n*Gender percentages:' # Gender percentages
f = user_list['USER_ID_hash'][user_list['SEX_ID']=='f'].unique().shape[0]
m = user_list['USER_ID_hash'][user_list['SEX_ID']=='m'].unique().shape[0]
print '% of female users: ', round(float(f)/(total_users),4)*100, '%'
print '% of male users: ', round(float(m)/(total_users),4)*100, '% \n'

print '*Registration status:'
users_withdrew = user_list[user_list['WITHDRAW_DATE'].isnull() == False]
users_active = user_list[user_list['WITHDRAW_DATE'].isnull()]
print '% of active users in the User list: ', round(float(users_active.shape[0])/total_users,4)*100, '%'
print '% of inactive users in the User list: ', round(float(users_withdrew.shape[0])/total_users,4)*100, '%\n'

print '*Among active users:'
active_f = users_active['USER_ID_hash'][user_list['SEX_ID']=='f'].shape[0]
active_m = users_active['USER_ID_hash'][user_list['SEX_ID']=='m'].shape[0]
print '% of active female users: ', round(float(active_f)/(users_active.shape[0]),4)*100, '%'
print '% of active male users: ', round(float(active_m)/(users_active.shape[0]),4)*100, '%'
print '\n*Among inactive users:'
inactive_f = users_withdrew['USER_ID_hash'][user_list['SEX_ID']=='f'].shape[0]
inactive_m = users_withdrew['USER_ID_hash'][user_list['SEX_ID']=='m'].shape[0]
print '% of inactive female users: ', round(float(inactive_f)/(users_withdrew.shape[0]),4)*100, '%'
print '% of inactive male users: ', round(float(inactive_m)/(users_withdrew.shape[0]),4)*100, '%'

*Total # of users:  22873

*Gender percentages:
% of female users:  48.02 %
% of male users:  51.98 % 

*Registration status:
% of active users in the User list:  95.97 %
% of inactive users in the User list:  4.03 %

*Among active users:
% of active female users:  48.54 %
% of active male users:  51.46 %

*Among inactive users:
% of inactive female users:  35.57 %
% of inactive male users:  64.43 %

User Age¶

Next, I look at a few other basic distribution of different user profiles. This is a distribution of users' ages. The single largest user group is between 35 and 40, if we use each bin size as 5.

In [38]:

plt.figure(figsize=(13,6))
sns.distplot(user_list['AGE'],bins = range(0,100,5),kde=False)
plt.xticks(np.arange(min(range(0,100,5)), max(range(0,100,5))+1, 5.0))
plt.xlabel('Age')
plt.ylabel('# of Users')
plt.title('User Age Distribution')
plt.show()

Just because I'm curious, I also look at user age distributions for inactive user groups and active user groups, to see if there are any differences. While the distributions look different, this could be attributable to the smaller size of the inactive user group (5% vs 95%).

In [56]:

plt.figure(figsize=(17,4))
plt.subplot(1, 2, 1)
sns.distplot(users_active['AGE'],bins = range(0,100,5),kde=False)
plt.xticks(np.arange(min(range(0,100,5)), max(range(0,100,5))+1, 5.0))
plt.xlabel('Age')
plt.ylabel('# of Active Users')
plt.title('Active User Age Distribution')

plt.subplot(1, 2, 2)
sns.distplot(users_withdrew['AGE'],bins = range(0,100,5),kde=False)
plt.xticks(np.arange(min(range(0,100,5)), max(range(0,100,5))+1, 5.0))
plt.xlabel('Age')
plt.ylabel('# of Inactive Users')
plt.title('Inactive User Age Distribution')
plt.show()

User Registration¶

It would be interesting to also look at purchasing patterns of each group (active vs. inactive) to see if inactive users purchase differenly or something causes them to cancel their status. For now, I will continue looking at registration characteristics by user.

This histogram below shows that most users signed up in 2011 (60% of users). This makes wonder how this dataset was chosen, and if it was random, what led such a spike in 2011 and whether that would affect further purchasing behaviors.

In [102]:

plt.figure(figsize=(10,6))
sns.distplot(user_list['REG_DATE'].dt.year,bins=3, kde=False)
plt.ticklabel_format(useOffset=False)
plt.xticks(np.arange(min(user_list['REG_DATE'].dt.year), max(user_list['REG_DATE'].dt.year)+1, 1.0))
plt.xlabel('Year')
plt.ylabel('# of Users who registered')
plt.title('Registration hour by user')
plt.show()

The two histograms below shows what time of the year and what time of the day users are signing up for registration.

November and May show the highest percentage of user registration rate. Throughout the day, registration starts to go up in the morning and peaks at 5pm and again after 9pm. The histogram on the right has KDE(kernel density estimate) turned on, so the y-axis scale shows density, rather than number of users.

In [69]:

plt.figure(figsize=(17,4))
#plot user registration by month
plt.subplot(1, 2, 1)
sns.distplot(user_list['REG_DATE'].dt.month,bins=12, kde=False)
plt.xlabel('Month')
plt.ylabel('# of Users who registered')
plt.title('Registration month by user')
plt.xticks(np.arange(min(user_list['REG_DATE'].dt.month), max(user_list['REG_DATE'].dt.month)+1, 1.0))

plt.subplot(1, 2, 2)
sns.distplot(user_list['REG_DATE'].dt.hour, kde=True)
plt.xlabel('Hour')
plt.ylabel('# of Users who registered')
plt.title('Registration hour by user')
plt.xticks(np.arange(min(user_list['REG_DATE'].dt.hour), max(user_list['REG_DATE'].dt.hour)+1, 1.0))
plt.show()

In looking at the below histogram by month and gender, there are no significant differences in gender for each month in registration. There seems to be the largest difference in December when there are more male users signing up than female users.

In [85]:

#plot user registration by month and gender

age = user_list.groupby(['REG_MONTH','SEX_ID']).size().unstack()
age.plot(kind = 'bar', colormap = cm.Accent, figsize=(13,6))
plt.ylabel('# of Users who registered')
plt.title('Registration month by user gender')
plt.show()

The histogram below is by registration hour and user gender. There are no significant gender differences by hour (male>female) with the largest difference in 5pm, which also happens to be the peak time for overall registration.

There are only 5 hours where females register more than males which are mostly late mornings and early afternoons (10am to 1pm).

In [92]:

hour = user_list.groupby(['REG_HOUR','SEX_ID']).size().unstack()
print '# of Hours(female > male): ', hour[hour['f']>hour['m']].shape[0]
print '\nDetails of these 5 hours:'
print hour[hour['f']>hour['m']]
print '\n# of Hours(male > female): ', hour[hour['m']>hour['f']].shape[0]
hour.plot(kind = 'bar', colormap = cm.Accent, figsize=(13,6) )
plt.ylabel('# of Users who registered')
plt.title('Registration hour by user gender')
plt.show()

# of Hours(female > male):  5

Details of these 5 hours:
SEX_ID      f    m
REG_HOUR          
1         311  309
10        548  518
11        658  637
12        708  639
13        592  568

# of Hours(male > female):  18

Ponpare.jp analytics intro (Series)

Sat, 26 Sep 2015 00:00:00 +0000

This is a first blog post of this marketing analytics series on a coupon e-commerce website (ponpare.jp).

Based on a year of transactional data published by a coupon e-commerce website (ponpare.jp) for this Kaggle contest, I plan to share some of the interesting marketing insights and trends in this series of blog posts. This blog posting covers a basic description of data background. In subsequent postings, I will cover topics such as product category specific patterns, customer classification through dimensionality reduction, retrospective A/B testing based on customer visits and purchases.

Brief Introduction of Data:

There are 5 main files which contain each user profile info, visit logs, transaction data and each coupon info. Some of these are divided between a training set and a test set for validation purposes as the goal of the Kaggle contest is making predictions. Depending on the type of analysis, I will clarify if I will be using one or both.

Total # of users (User List): 22,873
Training set: 2011-07-01 to 2012-06-23
Test set: 2012-06-24 to 2012-06-30

Structure of Files:

The variables represented below are simplified to show the basic characteristic of the dataset and relationships between each file.

User List (by each User_ID)
- User_ID
- Sex
- Gender
- etc
Coupon Visit (visit log, shows each visit by a user for any coupon)
- User_ID
- Coupon_ID
- Purchased? (1 or 0)
- etc
Coupon Detail (by each transaction)
- User_ID
- Coupon_ID
- Purchase_ID
- etc
Coupon List
- Coupon_ID
- Category name
- etc
Coupon Area (area in Japan where the coupon is listed)

Index of this Series:

Intro (this post)
Uncovering User Data
Purchasing patterns

Story of crime and education

Wed, 16 Sep 2015 00:00:00 +0000

The bubble chart below shows the relationships between literacy and homicide rates by country income groups in 2012.

In general, most countries are clustered around the lower part of the homicides but scattered throughout literacy rate. High income (both OECD and nonOECO) countries are in the uppermost left corner, indicating high literacy rate and low homicides. They have the least dispersion on both axes.

On the other hand, lower middle income and upper middle income countries have the most dispersion throughout the x-axis, the homicides. Low income countries have most dispersion throughout the y-axis, but most are at the lower end of the homicides.

Crime and Education in 2012

Bubble chart of Literacy rates and Homicides by Country in 2012

Data Source: http://data.worldbank.org

Miri's Blog