Author: admin

Do you want to become a ‘Data Scientist’? If yes, then the first step is to understand the basic terms and their usage.

A – Brief History

Data Science is not a new field as the statisticians were doing the job even before the computer invention. Though, the evolution of modern computing technologies empowered statisticians to solve a wide variety of practical problems with heavy number crunching and massive data storage. The terms ‘knowledge discovery’ and ‘data mining’ came widely in use in the late 1980’s after the invention of the database management system and the relational database management system. Later ‘Big data’ term published in the ACM Digital Library in 1997 after the database industry noticed the explosion of business data. In the late 1990’s, the term ‘Data Science’ inspired researchers and professionals and interchangeably replaced the word ‘statistician’.

B- Basic Concept

I- Big Data, Data Science & Machine Learning

Any data with three V’s i.e. Volume, Variety and Velocity is considered as Big Data. Big Data can’t be handled with conventional ways of data analysis and processing. Data science deals with Big Data and brings out meaningful insights. Due to its large scale, Data Science now depends on algorithms that try numerous possibilities to provide the best solution, here comes the Machine Learning.

II- Data Mining & Data Analytics

Machine Learning acts as a tool to identify unknown patterns in the Big Data and the process is called Data Mining, unlike Data analytics where the process starts with a specific hypothesis.

III- Big Data Analytics

The approach to breaking down a task into smaller pieces and assigned to different processors which could be geographically dispersed is called ‘Distributed Computing’. Big data analytics leverages distributed computing technologies to overcome computational challenges.

C- Technologies that Enable Data Science Into Reality

– Data Infrastructure: It supports data sharing, processing, and consumption. Distributed computing and cloud computing is most popular these days.

– Data Management: DBMS plays an important role to store structured and unstructured data sets. Since a majority of business-related data is structured, SQL knowledge is still invaluable.

– Visualization: It is very important to communicate newly acquired insights to the leadership and rest of the organization so data visualization technologies play an equally important role.

D- Data Science Applications

Data Science can be applied where ever ‘Big Data’ is involved. Following are only a few examples:

* Fraud detection
* Social Media Analytics
* Online matchmaking or dating services
* Weather forecast
* Simulation
* Network Security…etc.

E- Must Have Skills for Data Scientist

I- Statistics

Developing a reasonable understanding of statistics is a must for a data scientist as it lays the foundation of data science. At a minimum, a data scientist needs to be proficient with concepts such as probability, correlation, variables, distributions, regression, null hypothesis significance tests, confidence intervals, t-test, ANOVA, and chi-square. At an advanced stage, Data Scientist needs concepts and algorithms such as logistic regression, support vector machines (SVMs) and Bayesian method. Common statistical analysis tools such as Excel, R and SAS are very famous among Data Scientist.

II- Data mining

* Classification – Labelling a group of data objects into a specific category.
* Prediction – Building a model that produces continuous or ordered values that form a trend.
* Clustering – Grouping similar data objects into a class…etc.

* Natural Language Processing – NLP refers to different ways for a computer to interact with humans through a natural language. Computer science, Artificial Intelligence (AI), Computer linguistics and Human-computer interaction (HCI) are different areas of NLP. Some of the NLP aspects which are specifically related to Data Science are Tokenization, parsing, sentence, segmentation and named entity recognition. Python programming language is very famous and a recommended tool for having well-developed NLP tools.

* Tokenization and Parsing: Isolate each symbol from a text and conduct a grammatical analysis
* Sentence segmentation: Separates one sentence from the other in a text.
* Named entity recognition: Identifies which text symbol maps to what types of proper names

* Machine Learning (Supervised & Unsupervised)
* Visualization – Softwares are already available in the market that offers comprehensive visualization tools for data scientist such as Tableau. But it is important to remember that Data Scientist always acts as a middleman between data pile up and decision makers.

F- Roles and Responsibilities

Data Scientist or Engineer

A data scientist can work in any organization who is having data and willing to analyze its performance and future prediction. The role is more of a generalist instead of a specialist. A data scientist works with other data science specialist such as machine learning specialist.

Machine Learning Specialist

It’s a highly creative and independent role where you need the discipline to follow through and meet deadlines. Paying attention to details and quality is critical. Math and IT skills are essential as they form the foundations of the machine learning scientist. Deep knowledge of statistics and probability, ability to develop and validate a mathematical model, translating a model into an algorithm, proficiency in the programming language (Python, C++, Java, R…etc.), understanding of distributed computing are essential skills for a Machine Learning Specialist.

G- Related Certifications

* MCSE Business Intelligence Certification
* Cloudera Certified Professional or CCP data scientist
* Cloudera Certified Developer for Apache Hadoop or CCDH
* Cloudera Certified Administrator for Apache Hadoop or CCAH
* Cloudera Certified Specialist in Apache HBase or CCSHB
* EMC Data Science Associate (EMCDSA)
* EMC Data Center Architect or EMCDCA
* EMC Cloud Architect or EMCCA
* Oracle BI Implementation Specialist …etc.

H- Final Words

* Data Scientist must keep refreshing their knowledge to stay up to date. Attending conferences, workshops, peer networking and continuing education are ways to stay updated.
* Cloud vendors like Amazon, IBM, and Google …etc. makes it cheaper for companies to use cloud computing facilities instead of private in housed resources, which in turn increases the demand for Data Scientists. Even Data Scientist no longer worries about data infrastructure and management problems due to emerging online services.
* The importance of Machine Learning is growing especially deep learning taking advantage of neural networking is getting more traction.

For reference and details visit: First Step To Become a Data Scientist https://goo.gl/2qRo5w #DataScience #Cloud

Reviving from the dead an old but popular blog on Understanding Type I and Type II Errors

I recently got an inquiry that asked me to clarify the difference between type I and type II errors when doing statistical testing.  Let me use this blog to clarify the difference as well as discuss the potential cost ramifications of type I and type II errors. I have also provided some examples at the end of the blog[1]. 

In statistical test theory, the notion of statistical error is an integral part of hypothesis testing. The statistical test requires an unambiguous statement of anull hypothesis (H0), for example, “this person is healthy”, “this accused person is not guilty” or “this product is not broken”.   The result of the test of the null hypothesis may be positive(healthy, not guilty, not broken) or may be negative(not healthy, guilty, broken).

If the result of the test corresponds with reality, then a correct decision has been made (e.g., person is healthy and is tested as healthy, or the person is not healthy and is tested as not healthy).  However, if the result of the test does not correspond with reality, then two types of error are distinguished: type I errorand type II error.

Type I Error (False Positive Error)

A type I error occurs when the null hypothesisis true, but is rejected.  Let me say this again, atype I error occurs when the null hypothesis is actually true, but was rejected as falseby the testing.

A type I error, or false positive, is asserting something as true when it is actually false.  This false positive error is basically a “false alarm” – a result that indicates a given condition has been fulfilled when it actually has not been fulfilled (i.e., erroneously a positive result has been assumed).

Let’s use a shepherd and wolf example.  Let’s say that our null hypothesis is that there is “no wolf present.”  A type I error (or false positive) would be “crying wolf” when there is no wolf present. That is, the actual conditionwas that there was no wolf present; however, the shepherd wrongly indicated there was a wolf present by calling “Wolf! Wolf!”  This is a type I error or false positive error.

Type II Error (False Negative)

A type II error occurs when the null hypothesis is false, but erroneously fails to be rejected.  Let me say this again, atype II error occurs when the null hypothesis is actually false, but was accepted as trueby the testing.

A type II error, or false negative, is where a test result indicates that a condition failed, while it actually was successful.   A Type II error is committed when we fail to believe a true condition.

Continuing our shepherd and wolf example.  Again, our null hypothesis is that there is “no wolf present.”  A type II error (or false negative) would be doing nothing (not “crying wolf”) when there is actually a wolf present.  That is, the actual situationwas that there was a wolf present; however, the shepherd wrongly indicated there was no wolf present and continued to play Candy Crush on his iPhone.  This is a type II error or false negative error.

A tabular relationship between truthfulness/falseness of the null hypothesis and outcomes of the test can be seen in the table below:

 

Null Hypothesis is true

Null hypothesis is false

Reject null hypothesis

Type I Error

False Positive

Correct Outcome

True Positive

Fail to reject null hypothesis

Correct outcome

True Negative

Type II Error

False Negative

Examples

Let’s walk through a few examples and use a simple form to help us to understand the potential cost ramifications of type I and type II errors.  Let’s start with our shepherd / wolf example.

Null Hypothesis

Type I Error / False Positive

Type II Error / False Negative

Wolf is not present

Shepherd thinks wolf is present (shepherd cries wolf) when no wolf is actually present

Shepherd thinks wolf is NOT present (shepherd does nothing) when a wolf is actually present

Cost Assessment

Costs (actual costs plus shepherd credibility) associated with scrambling the townsfolk to kill the non-existing wolf

Replacement cost for the sheep eaten by the wolf, and replacement cost for hiring a new shepherd

Note: I added a row called “Cost Assessment.”  Since it can not be universally stated that a type I or type II error is worse (as it is highly dependent upon the statement of the null hypothesis), I’ve added this cost assessment to help me understand which error is more “costly” and for which I might want to do more testing.

Let’s look at the classic criminal dilemma next.  In colloquial usage, a type I error can be thought of as "convicting an innocent person” and type II error “letting a guilty person go free”.

Null Hypothesis

Type I Error / False Positive

Type II Error / False Negative

Person is not guilty of the crime

Person is judged as guiltywhen the person actually did notcommit the crime (convicting an innocent person)

Person is judged not guiltywhen they actually didcommit the crime (letting a guilty person go free)

Cost Assessment

Social costs of sending an innocent person to prison and denying them their personal freedoms (which in our society, is considered an almost unbearable cost)

Risks of letting a guilty criminal roam the streets and committing future crimes

Let’s look at some business related examples.  In these examples I have reworded the null hypothesis, so be careful on the cost assessment.

Null Hypothesis

Type I Error / False Positive

Type II Error / False Negative

Medicine A cures Disease B

(H0 true, but rejected as false)

Medicine A curesDisease B, but is rejected as false

(H0 false, but accepted as true)

Medicine A does not cureDisease B, but is accepted as true

Cost Assessment

Lost opportunity cost for rejecting an effective drug that could cure Disease B

Unexpected side effects (maybe even death) for using a drug that is not effective

Let’s try one more.

Null Hypothesis

Type I Error / False Positive

Type II Error / False Negative

Display Ad A is effective in driving conversions

(H0 true, but rejected as false)

Display Ad A is effective in driving conversions, but is rejected as false

(H0 false, but accepted as true)

Display Ad A is not effective in driving conversions, but is accepted as true

Cost Assessment

Lost opportunity cost for rejecting an effective Display Ad A

Lost sales for promoting an ineffective Display Ad A to your target visitors

The cost ramifications in the Medicine example are quite substantial, so additional testing would likely be justified in order to minimize the impact of the type II error (using an ineffective drug) in our example.  However, the cost ramifications in the Display Ad example are quite small, for both the type I and type II errors, so additional investment in addressing the type I and type II errors is probably not worthwhile.

Summary

Type I and type II errors are highly depend upon the language or positioning of the null hypothesis. Changing the positioning of the null hypothesis can cause type I and type II errors to switch roles.

It’s hard to create a blanket statement that a type I error is worse than a type II error, or vice versa.  The severity of the type I and type II errors can only be judged in context of the null hypothesis, which should be thoughtfully worded to ensure that we’re running the right test. 

I highly recommend adding the “Cost Assessment” analysis like we did in the examples above.  This will help identify which type of error is more “costly” and identify areas where additional testing might be justified.

[1]More information about type I and type II errors can be found at: http://en.wikipedia.org/wiki/Type_I_and_type_II_errors https://goo.gl/brxVgH #DataScience #Cloud