Day: January 3, 2018

ICYDK: I’m preparing to publish a bunch of transit-related data as APIs, for us across a number of applications from visualizations to conversation interfaces like bots and voice-enablement. As I’m learning about the data, publishing it as unsophisticated CRUD APIs, I’m thinking deeply about how I would enable others to ask questions of this data using web APIs. I’m thinking about the hard work of deriving visual meaning from specific questions, all the way to how would you respond to an Alexa query regarding transit data in less than a second, going well beyond what CRUD gives us when we publish our APIs and taking things to the next level.

Knowing the technology sector, the first response I’ll get is machine learning! You take all your data, and you train up some machine learning models, put some natural language process to work, and voila, you have your answer to how you provide answers. I think this is a sensible approach to many data sets, and for organizations who have the machine learning skills and resources at their disposal. There are also a growing number of SaaS solutions for helping put machine learning work to answer complex questions that might be asked of large databases. Machine learning is definitely part of the equation for me, but I’m not convinced it is the answer in all situations, and it might not always yield the correct answers we are always looking for. https://goo.gl/L86Z3h #DataIntegration #ML

ICYMI: I spend a lot of time futzing with editors and IDEs. To be honest, I’m pretty compulsive obsessive about the whole thing. I can watch YouTube videos for hours studying how each works and why people like them. One question that I really wanted to tackle is “Are there ways in which VS Code is actually better (i.e. more productive) than WebStorm, PyCharm, IntelliJ, etc." 

I think this video Moving from WebStorm to VSCode does a pretty good job tackling this question, but I remain unconvinced.  https://goo.gl/4tYxFq #DataIntegration #ML

ETL refers to extract, transform, load, and it is generally used for data warehousing and data integration. ETL is a product of the relational database era and it has not evolved much in the last decade. With the arrival of new cloud-native tools and platforms, ETL is becoming obsolete. There are several emerging data trends that will define the future of ETL in 2018. A common theme across all these trends is to remove the complexity by simplifying data management as a whole. In 2018, we anticipate that ETL will either lose relevance or the ETL process will disintegrate and be consumed by new data architectures.

Unified Data Management Architecture

A unified data management (UDM) system combines the best of data warehouses, data lakes, and streaming without expensive and error-prone ETL. It offers reliability and performance of a data warehouse, real-time and low-latency characteristics of a streaming system, and scale and cost-efficiency of a data lake. More importantly, UDM utilizes a single storage backend with benefits of multiple storage systems which avoids moving data across systems hence avoiding data duplication and data consistency issues. Overall, it creates less complexity to deal with. https://goo.gl/uDD7u1 #DataIntegration #ML