✨ If you missed the YC Demo Day a couple of weeks ago, you still have a chance to check out some incredible AI-driven tech in our AIPI Deep Learning Demo Day on April 23rd (12-2p EST). DM me for an invite! ✨
🗞 This Week in News
The AI Index Report - The AI Index report tracks, collates, distills, and visualizes data related to artificial intelligence (AI). The mission is to provide unbiased, rigorously vetted, broadly sourced data in order for policymakers, researchers, executives, journalists, and the general public to develop a more thorough and nuanced understanding of AI. The link includes the top 10 takeaways in addition to the full report. From Stanford University.
🥁 Interesting Products & Features
GPT-4 Turbo brings OpenAI [back] to the top of the leaderboards, with improved capabilities in writing, math, logical reasoning, and coding. Now available to paid ChatGPT users.
Cohere’s Rerank 3 - reranking model purpose-built to enhance search and RAG systems. They claim it can boost performance of running RAG applications with negligible impact to latency. Impressive results across many different types of applications shared in the article.
Mixtral releases 8x22B LLM - pre-trained generative sparse MoE model.
Reka Core - multimodal language model from Reka. Its three models allow image, video, and audio input in addition to text. Fun fact: The current number of employees at Reka is 20.
🚀 Cool New Product: Limitless AI - a small wearable designed to record and summarize your life.
📄 Interesting Papers
Curiosity-driven red-teaming for large language models - The authors developed a technique to train a red-team large language model to automatically generate diverse prompts that trigger a wider range of undesirable responses from the chatbot being tested. For their reinforcement learning approach, they utilized a technique called curiosity-driven exploration, where the model is incentivized to be curious about the consequences of each prompt it generates, so it will try prompts with different words, sentence patterns, or meanings. The model successfully provokes toxic responses from a LLaMA2 model that has been heavily fine-tuned using human preferences to avoid toxic outputs. Authors from MIT.
Exploring Concept Depth: How Large Language Models Acquire Knowledge at Different Layers? - This paper shows quantitatively that simpler concepts are learned in shallower layers and more complex tasks may only be discernible at deeper layers, if at all. They also provide code “Concept Depth” to measure where a concept is learned by LLMs of different sizes. Authors from Rutgers University.
SplatPose & Detect: Pose-Agnostic 3D Anomaly Detection - This paper proposes a novel 3D Gaussian splatting-based framework “SplatPose” which, given multi-view images of a 3D object, accurately estimates the pose of unseen views in a differentiable manner, and detects anomalies in them. Achieves SOTA results in both training and inference speed and detection performance. Authors from the L3S Research Center.
Large-scale Training of Foundation Models for Wearable Biosignals - This study employed self-supervised learning using the unlabeled sensor data collected under informed consent from the large longitudinal Apple Heart and Movement Study (AHMS) to train foundation models for two common biosignals: photoplethysmography (PPG) and electrocardiogram (ECG) recorded on Apple Watch. (~141K participants spanning ~3 years). The first study that builds foundation models using large-scale PPG and ECG data collected via wearable consumer devices. Authors from Apple.
🧠 Sources of Inspiration
BabyLM Challenge - Competition to see who can train the best multimodal models while using only as much data as the average human baby sees (<~10M tokens). Results due in September.