Tag Archives: DevOps
Program Synthesis with Large Language Models
This paper explores the limits of the current generation of large language models for program synthesis in general purpose programming languages. We evaluate a collection of such models (with between 244M and 137B parameters) on two new benchmarks, MBPP and MathQA-Python, in both the few-shot and fine-tuning regimes. Our benchmarks are designed to measure the ability of these models to synthesize short Python programs from natural language descriptions. The Mostly Basic Programming Problems (MBPP) dataset contains 974 programming tasks, designed to be solvable by entry-level programmers. The MathQA-Python dataset, a Python version of the MathQA benchmark, contains 23914 problems that evaluate the ability of the models to synthesize code from more complex text. On both datasets, we find that synthesis performance scales log-linearly with model size. Our largest models, even without finetuning on a code dataset, can synthesize solutions to 59.6% of the problems from MBPP using few-shot learning with a well-designed prompt. Fine-tuning on a held-out portion of the dataset improves performance by about 10 percentage points across most model sizes. On the MathQA-Python dataset, the largest fine-tuned model achieves 83.8% accuracy. Going further, we study the model’s ability to engage in dialog about code, incorporating human feedback to improve its solutions. We find that natural language feedback from a human halves the error rate compared to the model’s initial prediction. Additionally, we conduct an error analysis to shed light on where these models fall short and what types of programs are most difficult to generate. Finally, we explore the semantic grounding of these models by fine-tuning them to predict the results of program execution. We find that even our best models are generally unable to predict the output of a program given a specific input. Read More
#devops, #nlpHow to Hack APIs in 2021
Baaackkk iiin myyy dayyyyy APIs were not nearly as common as they are now. This is due to the explosion in the popularity of Single Page Applications (SPAs). 10 years ago, web applications tended to follow a pattern where most of the application was generated on the server-side before being presented to the user. Any data that was needed would be gathered directly from a database by the same server that generates the UI.
Many modern web applications tend to follow a different model often referred to as an SPA (Single Page Application). In this model there is typically an API backend, a JavaScript UI, and database. The API simply serves as an interface between the webapp and the database. All requests to the API are made directly from the web browser.
This is often a better solution because it is easier to scale and allows more specialised developers to work on the project, i.e. frontend developers can work on the frontend while backend developers work on the API. These apps also tend to feel snappier because page loads are not required for every request.
… All this to say – there are APIs everywhere now, so we should know how to hack and secure them. Read More
OpenAI can translate English into code with its new machine learning software Codex
AI research company OpenAI is releasing a new machine learning tool that translates the English language into code. The software is called Codex and is designed to speed up the work of professional programmers, as well as help amateurs get started coding.
In demos of Codex, OpenAI shows how the software can be used to build simple websites and rudimentary games using natural language, as well as translate between different programming languages and tackle data science queries. Users type English commands into the software, like “create a webpage with a menu on the side and title at the top,” and Codex translates this into code. The software is far from infallible and takes some patience to operate, but could prove invaluable in making coding faster and more accessible. Read More
Eden AI launches platform to unify ML APIs
Large cloud vendors like Amazon, Google, Microsoft, and IBM offer APIs enterprises can use to take advantage of powerful AI models. But comparing these models — both in terms of performance and cost — can be challenging without thorough planning. Moreover, the siloed nature of the APIs makes it difficult to unify services from different vendors into a single app or workflow without custom engineering work, which can be costly.
These challenges inspired Samy Melaine and Taha Zemmouri to found Eden AI (previously AI-Compare) in 2020. The platform draws on AI APIs from a range of sources to allow companies to mix and match models to suit their use case. Eden AI recently launched what it calls an AI management platform, which the company claims simplifies the use — and integration — of various models for end users. Read More
New toolkit aims to help teams create responsible human-AI experiences
Microsoft has released the Human-AI eXperience (HAX) Toolkit, a set of practical tools to help teams strategically create and responsibly implement best practices when creating artificial intelligence technologies that interact with people.
The toolkit comes as AI-infused products and services, such as virtual assistants, route planners, autocomplete, recommendations and reminders, are becoming increasingly popular and useful for many people. But these applications have the potential to do things that aren’t helpful, like misunderstand a voice command or misinterpret an image. In some cases, AI systems can demonstrate disruptive behaviors or even cause harm. Read More
Machine Learning Engineering for Production (MLOps)
GitHub Copilot: Your AI pair programmer
GitHub Copilot uses OpenAI technology to suggest lines and functions, as well as ways to write tests, and discover new APIs. GitHub says it works bestl for JavaScript, Python, TypeScript, Go, and Ruby. The capability is powered by OpenAI Codex, which was trained on a large concentration of public source code, making it more powerful than GPT-3’s code generator. Read More
#devopsMLOps: Comprehensive Beginner’s Guide
MLOps, AIOps, DataOps, ModelOps, and even DLOps. Are these buzzwords hitting your newsfeed? Yes or no, it is high time to get tuned for the latest updates in AI-powered business practices. Machine Learning Model Operationalization Management (MLOps) is a way to eliminate pain in the neck during the development process and delivering ML-powered software easier, not to mention the relieving of every team member’s life.
Let’s check if we are still on the same page while using principal terms. Disclaimer: DLOps is not about IT Operations for deep learning; while people continue googling this abbreviation, it has nothing to do with MLOps at all. Next, AIOps, the term coined by Gartner in 2017, refers to the applying cognitive computing of AI & ML for optimizing IT Operations. Finally, DataOps and ModelOps stand for managing datasets and models and are part of the overall MLOps triple infinity chain Data-Model-Code.
While MLOps seems to be the ML plus DevOps principle at first glance, it still has its peculiarities to digest. We prepared this blog to provide you with a detailed overview of the MLOps practices and developed a list of the actionable steps to implement them into any team. Read More
Rick's Cafe AI 