Large language models (LLMs) have achieved remarkable success in NLP and multimodal tasks. Despite these successes, their development faces two main challenges: (i) high computational cost; and (ii) difficulty in conducting fair and objective evaluations. LLMs are prohibitively expensive, making it feasible for only a few major players to undertake their training, thereby constraining both research and application opportunities. This underscores the importance of cost-effective LLM training. In this paper, we utilize a growth strategy to significantly reduce LLM training cost. We demonstrate that an LLM with 101B parameters and 0.31TB tokens can be trained on a 100K budget. We also adopt a systematic evaluation paradigm for the IQ evaluation of LLMs, in complement to existing evaluations that focus more on knowledge-oriented abilities. We introduce our benchmark including evaluations on important aspects of intelligence including symbolic mapping, rule understanding, pattern mining, and anti-interference. Such evaluations minimize the potential impact of memorization. Experimental results show that our model FLM-101B, trained with a budget of $100K, achieves comparable performance to powerful and well-known models, e.g., GPT-3 and GLM-130B, especially in the IQ benchmark evaluations with contexts unseen in training data. The checkpoint of FLM-101B will be open-sourced at this https URL. — Read More
#nlp, #strategyMonthly Archives: September 2023
IBM rolls out new generative AI features and models
Fighting for relevance in the growing — and ultra-competitive — AI space, IBM this week introduced new generative AI models and capabilities across its recently launched Watsonx data science platform.
The new models, called the Granite series models, appear to be standard large language models (LLMs) along the lines of OpenAI’s GPT-4 and ChatGPT, capable of summarizing, analyzing and generating text. IBM provided very little in the way of details about Granite, making it impossible to compare the models to rival LLMs — including IBM’s own. But the company claims that it’ll reveal the data used to train the Granite series models, as well as the steps used to filter and process that data, ahead of the models’ availability in Q3 2023. — Read More
Building Enduring Enterprise AI Products
… We wondered if generative AI (genAI) adoption would mirror the promise of the cloud back in 2010. The transition to cloud ended up requiring significant investment in infrastructure and time, not to mention change management. But the promise—and implementation—of AI is proving to be very different. — Read More
LLMs Are Not All You Need
Large Language Models (LLMs) are powering the next big wave of innovation in technology, as with the internet, smartphones, and the cloud — generative AI is poised to change the fabric of our society.
GenAI tools like GitHub Copilot have been supercharging the productivity of developers worldwide since 2021. … The way we work is soon to shift. Goldman Sachs expects GenAI to raise global GDP by 7% in the next ten years. …LLMs alone are good, but not 7% of global GDP good. We need the ecosystem built around LLMs to make the most of them. — Read More
LLMs, RAG, & the missing storage layer for AI
In the rapidly evolving landscape of artificial intelligence, Generative AI, especially Language Model Machines (LLMs) have emerged as the veritable backbone of numerous applications, from natural language processing and machine translation to virtual assistants and content generation. The advent of GPT-3 and its successors marked a significant milestone in AI development, ushering in an era where machines could not only understand but also generate human-like text with astonishing proficiency. However, beneath the surface of this AI revolution lies a crucial missing element, one that has the potential to unlock even greater AI capabilities: the storage layer. — Read More
Chinese social media campaigns are successfully impersonating U.S. voters, Microsoft warns
Chinese state-aligned influence and disinformation campaigns are impersonating U.S. voters and targeting political candidates on multiple social media platforms with improved sophistication, Microsoft said in a threat analysis report Thursday.
Chinese Communist Party-affiliated “covert influence operations have now begun to successfully engage with target audiences on social media to a greater extent than previously observed,” according to the report, which focused on the rise in “digital threats from East Asia.” — Read More
Time100 AI
Time has published its list of the 100 most influential people in AI.
What is unique about AI is also what is most feared and celebrated—its ability to match some of our own skills, and then to go further, accomplishing what humans cannot. AI’s capacity to model itself on human behavior has become its defining feature. Yet behind every advance in machine learning and large language models are, in fact, people—both the often obscured human labor that makes large language models safer to use, and the individuals who make critical decisions on when and how to best use this technology. Reporting on people and influence is what TIME does best. That led us to the TIME100 AI. — Read More
View the List
Spread Your Wings: Falcon 180B is here
Today, we’re excited to welcome TII’s Falcon 180B to HuggingFace! Falcon 180B sets a new state-of-the-art for open models. It is the largest openly available language model, with 180 billion parameters, and was trained on a massive 3.5 trillion tokens using TII’s RefinedWeb dataset. This represents the longest single-epoch pretraining for an open model.
You can find the model on the Hugging Face Hub (base and chat model) and interact with the model on the Falcon Chat Demo Space.
In terms of capabilities, Falcon 180B achieves state-of-the-art results across natural language tasks. It tops the leaderboard for (pre-trained) open-access models and rivals proprietary models like PaLM-2. — Read More
Pentagon unveils ‘Replicator’ drone program to compete with China
The Pentagon committed on Monday to fielding thousands of attritable, autonomous systems across multiple domains within the next two years as part of a new initiative to better compete with China.
The program, dubbed Replicator, was announced by Deputy Defense Secretary Kathleen Hicks, speaking at the National Defense Industrial Association’s Emerging Technologies conference here. — Read More
ImageBind: One Embedding Space To Bind Them All
We present ImageBind, an approach to learn a joint embedding across six different modalities – images, text, audio, depth, thermal, and IMU data. We show that all combinations of paired data are not necessary to train such a joint embedding, and only image-paired data is sufficient to bind the modalities together. ImageBind can leverage recent large scale vision-language models, and extends their zero-shot capabilities to new modalities just by using their natural pairing with images. It enables novel emergent applications ‘out-of-the-box’ including cross-modal retrieval, composing modalities with arithmetic, cross-modal detection and generation. The emergent capabilities improve with the strength of the image encoder and we set a new state-of-the-art on emergent zero-shot recognition tasks across modalities, outperforming specialist supervised models. Finally, we show strong few-shot recognition results outperforming prior work, and that ImageBind serves as a new way to evaluate vision models for visual and non-visual tasks. — Read More
Rick's Cafe AI 