Imagine having to go through 2.5GB of log entries from a failed software build — 3 million lines — to search for a bug or a regression that happened on line 1M. It’s probably not even doable manually! However, one smart approach to make it tractable might be to diff the lines against a recent successful build, with the hope that the bug produces unusual lines in the logs.
Standard md5 diff would run quickly but still produce at least hundreds of thousands candidate lines to look through because it surfaces character-level differences between lines. Fuzzy diffing using k-nearest neighbors clustering from machine learning (the kind of thing logreduce does) produces around 40,000 candidate lines but takes an hour to complete. Our solution produces 20,000 candidate lines in 20 min of computing — and thanks to the magic of open source, it’s only about a hundred lines of Python code. Read More
Tag Archives: DevOps
Decision points in storage for artificial intelligence, machine learning and big data
Artificial intelligence and machine learning storage is not one-size-fits-all. Analytics work differs, and has varied storage requirements for capacity, latency, throughput and IOPS. We look at key decision points. Read More
18 Handy Resources for Machine Learning Practitioners
Machine Learning is a diverse field covering a wide territory and has impacted many verticals. It is able to tackle tasks in language and image processing, anomaly detection, credit scoring sentiment analysis, forecasting alongside dozens of other downstream tasks. A proficient developer, in this line of work; has to be able to draw, borrow, and steal from many adjacent fields such as mathematics, statistics, programming, and most importantly common sense. I for one have drawn tremendous benefits from myriad of tools available to break down complex tasks into smaller more manageable components. It turns out that developing and training a model only takes a small fraction of the project duration. The bulk of the time and resources are spent on data acquisition, preparation, hyperparameter tuning, optimization, and model deployment. I have been successful in building a systematic knowledge base that has helped my team to tackle some common yet tough challenges. Read More
MLOps with a Feature Store
If AI is to become embedded in the DNA of Enterprise computing systems, Enterprises must first re-align their machine learning (ML) development processes to include data engineers, data scientists and ML engineers in a single automated development, integration, testing, and deployment pipeline. This blog introduces platforms and methods for continuous integration (CI), continuous delivery (CD), and continuous training (CT) with machine learning platforms, with details on how to do CI/CD machine learning operations (MLOps) with a Feature Store. We will see how the Feature Store refactors the monolithic end-to-end ML pipeline into a feature engineering and a model training pipeline. Read More
Deploy Your First Serverless AWS ML Solution Fast
I’ve been working with AWS SageMaker for a while now and have enjoyed great success. Creating and tuning models, architecting pipelines to support both model development and real-time inference, and data lake formation have all been made easier in my opinion. AWS has proven to be an all encompassing solution for machine learning use cases, both batch and real-time, helping me decrease time to delivery. Read More
#devops, #mlaasHow AI and Machine Learning are Evolving DevOps
The automation wave has overtaken IT departments everywhere making DevOps a critical piece of infrastructure technology. DevOps breeds efficiency through automating software delivery and allowing companies to push software to market faster while releasing a more reliable product. What is next for DevOps? We need to look no further than artificial intelligence and machine learning.
Most organizations quickly realize the promise of AI and machine learning, but often fail to understand how they can properly harness them to improve their systems. That isn’t the case with DevOps. DevOps has some natural deficiencies that are difficult to solve without the computing power of machine learning and artificial intelligence. They are key to advancing your digital transformation. Here are three areas where AI and machine learning are advancing DevOps. Read More
This Bot Hunts Software Bugs for the Pentagon
Late last year, David Haynes, a security engineer at internet infrastructure company Cloudflare, found himself gazing at a strange image. “It was pure gibberish,” he says. “A whole bunch of gray and black pixels, made by a machine.” He declined to share the image, saying it would be a security risk.
Haynes’ caution was understandable. The image was created by a tool called Mayhem that probes software to find unknown security flaws, made by a startup spun out of Carnegie Mellon University called ForAllSecure. Haynes had been testing it on Cloudware software that resizes images to speed up websites, and fed it several sample photos. Mayhem mutated them into glitchy, cursed images that crashed the photo processing software by triggering an unnoticed bug, a weakness that could have caused headaches for customers paying Cloudflare to keep their websites running smoothly. Read More
Velocity is the Most Dangerous Metric for Dev Teams
Agile Velocity is arguably the most popular software development metric in the world. It’s a very powerful metric when used for individual team sprint capacity planning. And there are two things we know about power… it comes with great responsibility and it corrupts. Read More
Corporate Tools for GPU Access and Software Development
Secure the software development lifecycle with machine learning
Every day, software developers stare down a long list of features and bugs that need to be addressed. Security professionals try to help by using automated tools to prioritize security bugs, but too often, engineers waste time on false positives or miss a critical security vulnerability that has been misclassified. To tackle this problem data science and security teams came together to explore how machine learning could help. We discovered that by pairing machine learning models with security experts, we can significantly improve the identification and classification of security bugs.
At Microsoft, 47,000 developers generate nearly 30 thousand bugs a month. These items get stored across over 100 AzureDevOps and GitHub repositories. To better label and prioritize bugs at that scale, we couldn’t just apply more people to the problem. However, large volumes of semi-curated data are perfect for machine learning. Since 2001 Microsoft has collected 13 million work items and bugs. We used that data to develop a process and machine learning model that correctly distinguishes between security and non-security bugs 99 percent of the time and accurately identifies the critical, high priority security bugs, 97 percent of the time. This is an overview of how we did it. Read More
Rick's Cafe AI 