Author: admin

ICYMI: I met up with an old grad school friend a few weeks back. He’s an accomplished professor of demography, basically the statistical study of populations, particularly surrounding births, deaths, and fertility. Where he once traveled extensively to conduct demographic surveys in remote parts of the world, my friend’s now content to work with available U.S. census data to develop models and test hypotheses. Like me, he’s a stats/techie guy accomplished in statistical packages such as SPSS, SAS, and Stata. He notes he’s played around with R but was warned early on it couldn’t scale for his demanding needs. We then morphed to an engaging discussion about open source and the evolution of statistical software. My sense was that he was wary of the R nay-Sayers, but found his statistical comfort zone hard to abandon.

Alas, though perhaps not appreciated in many statistical settings, the R of 2018 is quite a step up over the R of 2008, driven significantly by contributions such as tidyverse and data.table from its extensive open source ecosystem. Indeed, the R code I write today would be unrecognizable to a core R stickler of even 5 years ago. And as for R’s very real RAM limitation, the 64 GB RAM, 2 TB SSD computer I purchased 2 years ago for $2500 has handled 95% of the challenges I’ve thrown at it with aplomb. The belligerent 5% I take to the Cloud and Spark.

As our discussion progressed, we opted to “challenge” R with the type of work he now does in his research. We settled on the 5 year (2012-2016) ACS Public Use Microdata Sample files PUMS as our data source. The PUMS “are a sample of the actual responses to the American Community Survey and include most population and housing characteristics. These files provide users with the flexibility to prepare customized tabulations and can be used for detailed research and analysis. Files have been edited to protect the confidentiality of all individuals and of all individual households. The smallest geographic unit that is identified within the PUMS is the Public Use Microdata Area (PUMA).”. Tellingly, there’re freely-available R packages demography and censusapi that make life a lot easier for R demographers.

The data we used consists of a series of household and population files. Each household has one or more persons (population), while each person is for one and only one household. It turns out that there are about 7.4M households consisting of 15.7M persons represented in this sample. A common key, “serialno”, connects the data.tables. Our task was to load both household and population data, with the mandate of joining the two to build topic-oriented data sets on demand. The raw household and population data are downloaded from zip files consisting of 4 CVS’s each. 

The development environment consists of Jupyter Notebook with a Microsoft R kernel. R’s tidyverse, data.table, and fst packages drive the data build. What follows is the implementation code.

Read the entire blog here. https://goo.gl/ZeUBNt #DataScience #Cloud

 Summary:  To take advantage of data science, an organization needs to consider their data quality and accessibility, and the willingness of their staff to use the results of data analysis results.  Most importantly, an organization must have a clear understanding of how it expects to benefit from data science.

Can data science benefit your organization?  If so, is your organization ready to take advantage of it?

“Data science” has been receiving a lot of attention recently – according to LinkedIn’s 2017 U.S. Emerging Jobs Report, data scientist roles have grown over 650 percent since 2012.  This article presents the steps for assessing your organization’s readiness for data science, and what you can do to take full advantage of it.

File Cabinets Are Becoming Obsolete

A major transformation has occurred in the way organizations manage their data since the advent of affordable computing.  Most of the transactions conducted by businesses, government agencies, and non-profit organizations are “paperless”, meaning that they are recorded electronically.  Business transactions are often recorded at electronic cash registers or via on-line purchases, and the services provided by government and non-profit organizations are often entered directly into a computer at the point of contact with the client.  The transformation of data from paper to computers has made it possible to manage and analyze data in ways that were never before possible.

The Next Frontier – Data Quality

Although it is likely that your organization’s data is already digitalized, this does not automatically mean that it is ready for analysis. The first step in assessing an organization’s readiness for data science is a data audit.  This a review of all the potential data analysis fields for their completeness and validity. 

Completeness refers to the extent of missing values found within each field.  Many analytic procedures have a “listwise” setting for missing values, which means a missing value in any of the fields that are selected for a procedure can cause an entire record to be excluded from the analysis.  Excluding records from your analysis can seriously bias your results.  A rule of thumb is that excluding more than 10% of your records can bias your results, so a data audit should flag any field in which more than 10% of its values are missing.  These fields could be simply omitted from the data analysis, or the missing values could be “imputed”, i.e. replaced with valid values.  Imputation methods range in complexity from replacing the missing values with the mean of the valid values to more sophisticated algorithms based on the principles of regression.  

Validity refers to the accuracy of the data found within each field selected for analysis.  For fields that include codes representing categories, this means that these codes correctly represent categories relevant to your organization.  For instance, a satisfaction code of “6” is invalid if satisfaction levels were coded as 1 through 5.  For fields with continuous values, some values may be unusually large or small.  A typical definition of unusually large or small values, known as “outliers”, are those that are more than two or three standard deviations from the mean for a field.  Invalid categorical codes and outlier values should be corrected or converted to missing values, but continuous fields with valid outliers could be “coerced”.  Coercion converts any values beyond a specified range, such as two standard deviations from the mean, into the values at the end of that range.  This dampens the disproportional influence of outliers on an analysis without the risk of losing records due to missing values.

Are You and Your Staff Ready and Willing to Use the Results of Data Science?

Data science can be a disruptive process, and as Uber and AirBnB will argue, this can be a good thing.  However, it is not uncommon for staff to resist change, and relying on data analysis rather than staff experience and intuition can represent a major change.  For example, experienced sales staff may be confident that they can identify a good lead, and therefore be reluctant to use “propensity scores” that rank a lead’s likelihood to respond to a sales offer.  Be prepared to explain the potential value of data science to your staff well before its implementation.

In addition to ensuring that your staff are ready to take advantage of data science, you will also need to consider how your data will be accessed for analysis and how the results will be distributed.  Data science requires quick and ongoing access to your data, which may raise some challenges for your IT staff.  Distribution of the data science results may also require the acquisition of business intelligence software that allows users to “slice and dice” these results, i.e. use drop-down list to select only the results relevant to them.

Most importantly, you will need to carefully consider how data science will ultimately benefit your organization either in terms of increased revenue and decreased costs, or other ways that support your organization’s mission.  There are an increasing number of firms that can guide you through this assessment process.  The bottom line is that data science enables you to extract value from your organization’s electronic data, and it is well worth your time to explore how you can make this happen with your organization. https://goo.gl/GfSYgn #DataScience #Cloud