Day: November 7, 2018

This past Monday, on November 5th, 2018, Snap Inc., the parent company of Snapchat and Spectacles, unveiled that they teamed up with Comscore, a measurement company for cross-platform audiences.

This now allows third-party data availability for Snapchat Discover, a feature within the app where users go to view “stories” from multiple media companies, according to Adweek. https://goo.gl/LJ19pb

#ICYMI: AI and machine learning have advanced rapidly over the past few years, and many have suggested that 2019 will be the year for businesses that have waited to finally embrace this new technology. Only 15% of enterprises are currently using AI, but 31% are slated to add it to their strategy over the next 12 months, according to a Digital Trends Report by Adobe. There is arguably tremendous value for companies that utilize automation technologies for data and analytics, and the process of making machine learning models more precise for business isn’t just about technology, but also the roles of people in the technology sector. While many point to the rise of artificial intelligence and machine learning as the beginning stages of human obsolescence, they couldn’t be more incorrect in their assumptions. Human beings are, and will remain, the biggest factor in successful analytics. 

Those looking to enter the data analytics workforce can breathe a sigh of relief knowing that AI will not be taking their job. It will, however, become a vital skill required of many—if not all in the coming years. According to the Digital Trends Report, the share of jobs requiring AI has increased 450% since 2013. Every industry has seen an exponential increase in data—with a great many organizations quite literally drowning in their proverbial data lakes. Expert projections suggest this issue is only going to grow, with a 4,300% increase in annual data production that will create 35 zettabytes by 2020.

With ever-growing hordes of data, businesses are beginning to recognize the necessity for increased automation strategy across their organization. According to Narrative Science, 61% of companies with an innovation strategy are using AI to identify opportunities in data that they would have otherwise missed, and only 22% companies without this strategy have been able to utilize AI to identify opportunities they would have otherwise missed. This is a huge opportunity for businesses to capitalize on data and button up inefficiencies, yet the majority of employees believe the emergence of AI to be a threat to their livelihood, with two-thirds of Americans expecting that computers or robots will do most of the work performed by humans in 50 years, and a recent study found that these beliefs are so pervasive that they are making employees sick. 

The reality is that AI and machine learning amplify human intelligence through technology, and employees can rest easy knowing that according a study released by the World Economic Forum, data-related jobs will be the most in-demand within the next four to five years, along with AI and machine learning specialists. While technologists are still figuring out which business processes should be automated versus what can be automated, striking a proper balance between automation and human intelligence in data analytics is no easy task. A common mistake around automation is failing to recognize the necessity of the human element for the technology. While computers are certainly capable of extraordinary feats, human programming is essentially still the brain behind these outputs. Human intelligence is required to create any kind of automation right now, and the resulting automated processes are therefore not inherently intelligent themselves.

Despite this, there remains continued speculation about the various technological challenges before machines can cognitively match the performance of human counterparts, and whether machines will ultimately replace humans. Matching or exceeding human performance is only part of the conversation. None of these complex tools can exist without at least some initial human involvement, which is why all the discussion of automation and job displacement tends to be exaggerated.

In truth, since our earliest days as a species, we developed and lived alongside technology to create a better life. Why is this stage of development any different?

  https://goo.gl/SXZLzh

#ICYMI: K-means algorithm is a popular and efficient approach for clustering and classification of data. My first introduction to K-means algorithm was when I was conducting research on image compression. In this applications, the purpose of clustering was to provide the ability to represent a group of objects or vectors by only one object/vector with an acceptable loss of information. More specifically, a clustering process in which the centroid of the cluster was optimum for the cluster and the clusters were optimum with respect to the centroids.

The dimensionality of the vectors ranged from 4 to 70 and even higher in which each cluster of N-dimensional vectors was to be represented by a single vector while minimizing a certain fidelity criterion or loss of information. The design process consisted of using a training set of vectors and to apply the results for outside the training set vectors.

Here, naturally, a couple of questions arises. First, what kind of cost function, fidelity criterion or distortion measure should be used to represent the penalty that is to be paid by our clustering process and its representation by only a single vector.

Second, how many clusters should there be and how to assign the samples to each cluster. The number of clusters to be chosen is a difficult task. Of course, if you have as many clusters as there are samples or vectors, then you have achieved this minimization. But, then, that is not considered clustering of the data. In any practical application, you have to resort to representing all the samples/vectors with a few samples/vectors and therefore create a set of cluster of vectors.

Let’s discuss each of these questions.

Which Cost function, Distortion Measure or Fidelity Criterion?

The answer to the question would depend on your application. A widely used measure is the mean square error also referred to as L2-norm. The L2-norm is commonly used most applications including signal processing, image compression, and video compression. There are applications that use absolute error difference, the L1-norm, also known as Manhattan distance. There are many other error measures such as the one used for speech that uses a weighted distortion measure called the Itakura-Saito measure. The distortion measure indicates the penalty to be paid for representing the members of the clusters by a single vector. As a result, the centroid of the cluster, that represents the cluster itself, is the generalized center of gravity of the cluster. The center of gravity of the cluster refers to the mean value of all the vectors in the cluster when mean-square-error is used as the distortion measure. To generalize this to other distortion measures, we use the term generalized-center-of-gravity to represent the centroid of the cluster for other distortion measures.

The second question relates to how many clusters and how to assign samples to clusters? The answer to the second part of the question is easy since the assignment of vectors to clusters follows directly from the cost function used in your system. The number of clusters, however, is an important one and would be discussed later.

Now, how do we decide on the clusters and their representation? We may answer this question by using an optimization by iteration technique.

More specifically, in this approach, we continuously optimize

a) the clusters for the given centroids, and

b) the centroids for the clusters,

until an optimum partitioning of clusters and centroid assignment is reached.

We can represent this approach by the following algorithm.

Iterative Algorithm:

*

Initialization: Given the number of centroids N and their values A(m), a distortion threshold, e>0, and the points for all clusters, set iteration number m=0 and distortion value

D(-1)=Infinity.

*

Given the set of centroids, find the minimum distortion partition S(m) for the centroid set A(m) and compute the resulting distortion, D(m) ,based on distortion criterion.
*

If (D(m-1)-D(m))/D(m)