According to a Gartner study, 85 percent of AI projects will flatline by 2022. Even the most diligent machine learning models may not meet expectations when deployed in an enterprise setting, mainly due to two reasons — inadequate data infrastructure and talent scarcity.

In the machine learning pipeline, search for appropriate data and dataset preparation are among the most time-consuming processes. A data scientist spends around 80 percent of his/her time in managing and preparing data for analysis. The demand-supply gap for qualified data scientists is another pressing challenge.

Enter, feature store.  Read More

AWS has made available Amazon Lookout for Metrics, a service that uses machine learning (ML) to automatically monitor various metrics across business and operational data, detect anomalies and alert the user so they can take appropriate action.

According to AWS, Lookout for Metrics is based on technology used by Amazon itself in business operations, and so reflects 20 years of the firm’s experience in anomaly detection and machine learning. It was built to allow developers to set up autonomous monitoring of important metrics to detect anomalies and identify their root cause in a matter of few clicks. This, AWS claimed, would make it easier to diagnose the root cause of anomalies such as unexpected dips in revenue, high rates of abandoned shopping carts, spikes in payment transaction failures, or increases in new user sign-ups. Read More

Version control is used to keep track of modifications made in a software code. Similarly, when building machine learning (ML) systems, it is essential to track things, such as the datasets used to train the model, the hyperparameters and pipeline used, the version of tensorflow used to create the model, and many more.

ML artifacts’ history and lineage are very complicated than a simple, linear log. Git can be used to track the code to one extent, but we need something to track your models, datasets, and more. The complexity of ML code and artifacts like models, datasets, and much more requires a similar approach.

Therefore, the researchers have introduced Machine Learning Metadata (MLMD), a standalone library to track one’s entire ML workflow’s full lineage from data ingestion, data preprocessing, validation, training, evaluation, deployment, etc. MLMD also comes integrated with TensorFlow Extended.  Read More

Conway’s law is critical to understanding the forces at play when organizing teams amidst the long-lasting, unattended impact they can have on our software systems, as the latter have become larger and more interconnected than ever before. But you might wonder if a law from 1968 about software architecture has stood the test of time.

We’ve come a long way after all: microservices, the cloud, containers, serverless. Such novelties can help teams improve locally, but the larger the organization, the harder it becomes to reap the full benefits. The way teams are set up and interact is often based on past projects and/or legacy technologies (reflecting the latest org-chart design, which might be years old, if not decades).

This quote from Ruth Malan provides what could be seen as the modern version of Conway’s law: “If the architecture of the system and the architecture of the organization are at odds, the architecture of the organization wins.” Read More

LinkedIn today open-sourced Dagli, a machine learning library for Java (and other JVM languages) that ostensibly makes it easier to write bug-resistant, readable, modifiable, maintainable, and deployable model pipelines without incurring technical debt. Read More

#devops

The lines between the real world and the digital world have been consistently blurring for years, and with that, software has bloomed. Physicists are hypothesizing that information can be considered a form of matter, the fifth form of matter in fact.

More and more, software is linked to the quality of our lives. That means the quality of our software will fundamentally direct the quality of our experience, so there’s never been a more important time to seek out ways to improve our DevOps. One of the tools that helps us explore that is ML. Read More