Google's Open-Source Gemma Enters the Fray — Now Who's the "CloseAI"? | BlueRun Ventures

A few key points about Gemma: 1) It's a small large language model, available in 2B (2 billion parameters) and 7B (7 billion parameters) sizes, runnable on phones and PCs; 2) It allows all organizations to responsibly use and distribute it for commercial purposes, with a developer toolkit included — very friendly; 3) Open source, open source, and open source. Last night, a user on X mocked OpenAI: "Where's the 'Open'?" Here's today's share — enjoy.

The open-source LLM space has just welcomed a major new player.

Google has launched a new open-source model family called "Gemma." Compared to Gemini, Gemma is more lightweight while remaining free to use, with model weights fully open-sourced and commercial use permitted.

Gemma official page: https://ai.google.dev/gemma/

This release includes two weight-scale variants: Gemma 2B and Gemma 7B. Each size comes in both pretrained and instruction-tuned versions. Users can access them through Kaggle, Google's Colab Notebook, or Google Cloud.

Of course, Gemma also launched immediately on HuggingFace and HuggingChat, so anyone can test its generation capabilities:

Despite its smaller size, Google claims the Gemma models have "significantly outperformed much larger models on key benchmarks," including comparisons against Llama-2 7B and 13B, as well as the buzzy Mistral 7B.

Moreover, Gemma "can run directly on developers' laptops or desktop computers." Beyond the lightweight models, Google has also introduced tools to encourage collaboration and guidelines for responsible use.

François Chollet, creator of Keras, put it bluntly: the title of strongest open-source large model has changed hands.

On HuggingFace's LLM leaderboard, Gemma's 2B and 7B models have both reached the top spot.

The new Responsible Generative AI Toolkit provides guidance and essential tools for building safer AI applications with Gemma. Google also offers an inference and supervised fine-tuning (SFT) toolchain for Gemma through native Keras 3.0 compatibility with all major frameworks (JAX, PyTorch, and TensorFlow).

While major tech companies and AI research institutions explore hundred-billion-parameter multimodal models, many startups are also working to build language models in the tens-of-billions range. Meta's Llama series, launched last year, shook the industry and sparked debate over open versus closed approaches to generative AI.

Google says Gemma uses the same research and technology used to build the Gemini models. However, Gemma's direct entry into the open-source ecosystem is a dramatically different approach from Gemini. Google also appears to be abandoning its strategy from last year of "no longer opening up core technology."

While developers can build on Gemini, they must do so either through APIs or on Google's Vertex AI platform — seen as a closed model. Compared to OpenAI, which follows a similar closed-source path, this offered no clear advantage.

But with Gemma's open-source release, Google may be able to attract more users to its AI models rather than losing them directly to competitors like Meta and Mistral.

Google's surprise open-source drop may be an attempt to get ahead of Meta's Llama 3 by a day, given earlier reports that the Llama series would see a new release this week (we look forward to first-hand benchmark comparisons).

Though it's only been out for a few hours, X users have already begun sharing their experiences. One user reported that Gemma-7B is fast with stable output, outperforming Llama-2 13B.

Alongside the open-source models, Google also published a technical report detailing Gemma's performance, dataset composition, and modeling methodology. Other researchers found notable highlights in the report, such as Gemma's vocabulary size of 256K, meaning it can provide better and faster support for languages beyond English.

Here are the technical report details.

Gemma Technical Details

Overall, Gemma is a lightweight SOTA open model family demonstrating strong performance in language understanding, reasoning, and safety.

Technical report link: https://storage.googleapis.com/deepmind-media/gemma/gemma-report.pdf

Google released two versions of the Gemma model, with 2 billion and 7 billion parameters respectively, providing both pretrained checkpoints and those fine-tuned for dialogue, instruction following, helpfulness, and safety. The 7 billion parameter model is designed for efficient deployment and development on GPU and TPU, while the 2 billion parameter model targets CPU and on-device applications. The different sizes accommodate varying compute constraints, applications, and developer requirements.

Gemma outperforms open models of similar parameter scale on 11 of 18 text-based tasks, including question answering, commonsense reasoning, math and science, and coding.

Figure 1 below shows performance comparisons between Gemma (7B) and LLaMA 2 (7B), LLaMA 2 (13B), and Mistral (7B) on question answering, reasoning, math and science, and coding tasks. Gemma (7B) shows advantages (except in question answering, where it lags behind LLaMA 2 (13B)).

Next, let's look at Gemma's model architecture, training infrastructure, pretraining, and fine-tuning methods.

Model Architecture

The Gemma model architecture is based on the Transformer decoder. Table 1 summarizes the core parameters of this architecture. Models are trained with a context length of 8192 tokens.

Additionally, Google made improvements beyond the original transformer paper, including:

• Multi-query attention: The 7B model uses multi-head attention, while the 2B checkpoint uses multi-query attention;
• RoPE embeddings: Gemma uses rotary positional embeddings at every layer rather than absolute positional embeddings; Gemma also shares embeddings between input and output to reduce model size;
• GeGLU activations: Standard ReLU nonlinearity is replaced with GeGLU activation functions;
• Normalizer Location: Gemma normalizes both the input and output of each transformer sublayer, differing from standard practice of normalizing only one or the other, with RMSNorm as the normalization layer.

Training Infrastructure

Google used its self-developed AI chip TPUv5e to train the Gemma models: TPUv5e is deployed in pods of 256 chips, configured as a 2D torus of 16 x 16 chips.

For the 7B model, Google trained across 16 pods (4,096 TPUv5e total). They pretrained the 2B model across 2 pods, totaling 512 TPUv5e. Within a pod, Google used 16-way model sharding and 16-way data replication for the 7B model. For the 2B model, only 256-way data replication was needed. Optimizer states were further sharded using a ZeRO-3-like technique. Beyond the pod, Google used the Pathways approach to perform data replication reduction over the datacenter network.

Pretraining

Gemma 2B and 7B were trained on 2T and 6T tokens of primarily English data from web documents, mathematics, and code respectively. Unlike Gemini, these models are not multimodal and were not trained to achieve state-of-the-art performance on multilingual tasks.

For compatibility, Google used a subset of Gemini's SentencePiece tokenizer (Kudo and Richardson, 2018). It splits digits, does not remove extra whitespace, and follows the techniques used by Chowdhery et al. (2022) and the Gemini team (2023) for byte-level encoding of unknown tokens. The vocabulary size is 256k tokens.

Instruction Tuning

Google fine-tuned Gemma 2B and Gemma 7B models through supervised fine-tuning (SFT) on a mixture of text-only, English-only synthetic and human-generated prompt-response pairs, as well as reinforcement learning from human feedback (RLHF) using reward models trained on English-only labeled preference data and policies based on a set of high-quality prompts.

Experiments found that both the supervised fine-tuning and RLHF stages are important for improving performance on downstream automatic evaluations and human preference assessments of model outputs.

Supervised Fine-Tuning

Google selected its mixture data for supervised fine-tuning based on parallel evaluation results from language models. Given a set of held-out prompts, Google generated responses from the test model and from a baseline model for the same prompts, then asked a larger, high-capacity model to express its preference between the two responses.

Google also constructed different prompt sets to highlight specific capabilities such as instruction following, factuality, creativity, and safety. They used different automated LM "judges" employing various techniques including chain-of-thought prompting and alignment with human preferences.

Formatting

Instruction-tuned models are trained with a specific formatter that annotates all instruction-tuning examples with additional information during both training and inference. This serves two purposes: 1) indicating roles in a conversation, such as the user role; 2) describing conversation turns, especially in multi-turn dialogue. To achieve these purposes, Google reserved special control tokens in the tokenizer.

Table 3 below shows the relevant formatting control tokens, and Table 4 shows a conversation example.

Reinforcement Learning from Human Feedback (RLHF)

Google further fine-tuned the supervised fine-tuned models using RLHF, collecting preference pairs from human raters and training a reward function under the Bradley-Terry model, similar to Gemini. The policy was trained to optimize this reward function using a REINFORCE variant with a Kullback–Leibler regularization term against the initial tuned model.

As with the SFT stage, Google relied on high-capacity models as automatic evaluators for hyperparameter tuning and to additionally mitigate reward hacking, computing comparison results against baseline models.

Evaluation

Google comprehensively evaluated Gemma across broad domains using human preference, automatic benchmarks, and memorization metrics.

Human Preference Evaluation

Beyond running standard academic benchmarks on fine-tuned models, Google conducted human evaluation studies on final release candidate models for comparison against Mistral v0.2 7B Instruct.

Compared to Mistral v0.2 7B Instruct, Gemma 7B IT achieved a positive win rate of 51.7%, while Gemma 2B IT achieved 41.6%. On approximately 400 prompts testing basic safety protocols, Gemma 7B IT achieved a win rate of 58%, and Gemma 2B IT achieved 56.5%. Corresponding figures are reported in Table 5.

Automatic Benchmark Evaluation

Google also compared Gemma 2B and 7B models against several external open-source LLMs on a range of academic benchmarks, as shown in Table 6:

On MMLU, Gemma 7B outperformed all open-source models of the same or smaller scale, as well as several larger models including LLaMA2 13B.

However, benchmark authors assessed human expert performance at 89.8%, with Gemini Ultra being the first model to exceed this threshold. This shows Gemma still has considerable room for improvement to reach Gemini and human-level performance.

But Gemma models performed particularly strongly on math and coding benchmarks. On math tasks, Gemma models outperformed other models by at least 10 points on both GSM8K and the more difficult MATH benchmark. Similarly, they scored at least 6 points higher than other open-source models on HumanEval. Gemma even outperformed the code-fine-tuned CodeLLaMA-7B model on MBPP (CodeLLaMA scored 41.4% versus Gemma 7B's 44.4%).

Memorization Evaluation

Google tested Gemma's memorization capabilities using the method adopted by Anil et al., specifically sampling 10,000 documents from each corpus and using the first 50 tokens as prompts for the model. They focused primarily on verbatim memorization, classifying text as memorized if the model's generated subsequent 50 tokens exactly matched the true subsequent text in the document. Figure 2 compares evaluation results against same-scale PaLM and PaLM 2 models, with results shown below.

Private Data

For large models, the possibility of private data being memorized is a significant concern. To make Gemma pretrained models safe and reliable, Google used automated methods to filter certain private information and other sensitive data from the training set.

To identify potential private data, Google used Google Cloud Data Loss Prevention (DLP) tools. These tools output three severity levels based on categories of private data (such as names, emails, etc.). Google classified the highest severity as "sensitive" and the other two as "personal," then measured how many stored outputs contained sensitive or personal data.

As shown in Figure 3 below, Google observed no instances of stored sensitive data, but did find that Gemma models memorized some data classified as potentially "personal" under the above categories. Notably, the tools used in this study may have many false positives (as they match patterns without considering context), meaning experimental results may overestimate the amount of identified private data.

Regarding memorization data volume, as shown in Figure 4 below, Google observed approximately 50% more data being memorized, and this was largely consistent across each distinct subcategory of the dataset.

Finally, Google also evaluated Gemma's safety through standardized AI safety benchmarks, with results shown in Table 8 below.

Reference links:

https://www.theverge.com/2024/2/21/24078610/google-gemma-gemini-small-ai-model-open-source

▶ More Exciting Content ◀

Founded in Silicon Valley in 2005, BlueRun Ventures is a venture capital firm focused on early-stage startups.

Currently, BlueRun Ventures manages multiple USD and RMB dual-currency funds in China, with assets under management exceeding RMB 15 billion, making it one of the largest early-stage funds in the country. It invests primarily at Pre-A and Series A stages, covering technology, consumer, and healthcare sectors, with nearly 200 portfolio companies including Li Auto, Waterdrop, QingCloud, Guazi.com, Qudian, Ganji.com, Energy Monster, Gaussian Robotics, Songguo Mobility, Yuntu Semiconductor, Machenike, CloudSaints Intelligence, Anxin Network Shield, BioMap, and others.

BlueRun Ventures has been ranked #1 on Zero2IPO's "China's Top 30 Early-Stage Investment Institutions" and ChinaVenture's "China's Best Early-Stage Venture Capital Institutions TOP30," and was named one of Preqin's Top 10 VC fund managers globally for sustained high returns.

Additionally, BlueRun Ventures has received consecutive annual awards from Forbes China, 36Kr, Cyzone, Caixin Media, CBNweekly, Jiemian, and other media outlets for honors including "China's Best Early-Stage Institution of the Year," "China's Top Venture Capital Firm," "Early-Stage Institution Most Welcomed by Entrepreneurs," and "Most Influential Early-Stage Institution of the Year."