Linear Capital portfolio company Chaoduijie (超对称) partners with Fudan University to release and open-source the 12-billion-parameter language model BBT-2

Supersymmetry Technologies Releases and Open-Sources 12 Billion Parameter Language Model BBT-2

In May 2022, Supersymmetry Technologies released the first version of its large language model Big Bang Transformer [Qianyuan], BBT-1 — a 1 billion parameter pre-trained language model trained on Chinese financial corpora. After its release, it received enthusiastic feedback from clients in the economics and finance sectors, and has since served as a foundational model for multiple domestic and international institutions. Recently, Supersymmetry launched BBT-2, a 12 billion parameter general-purpose large language model, and built specialized models for code, finance, and text-to-image generation on top of BBT-2.

The development of the Big Bang Transformer [Qianyuan] 12B model was powered by NVIDIA DGX infrastructure. Supersymmetry led the pre-training, and collaborated with Fudan University's Knowledge Works Laboratory on instruction fine-tuning and evaluation. Supersymmetry Technologies will release a series of models based on BBT-2 (model index at https://bbt.ssymmetry.com):

• BBT-2-12B-Text: 12 billion parameter Chinese foundation model
• BBT-2.5-13B-Text: 13 billion parameter Chinese-English bilingual foundation model
• BBT-2-12B-TC-001-SFT: Instruction fine-tuned code model capable of dialogue
• BBT-2-12B-TF-001: Financial model trained on the 12 billion parameter base for financial domain tasks
• BBT-2-12B-Fig: Text-to-image model
• BBT-2-12B-Science: Scientific paper model

Additionally, through its open-source partnership with UCloud, Supersymmetry has open-sourced three large models on its official website, GitHub, and UCloud — users can now deploy these directly via UCloud's GPU cloud host industry images or compute platform:

• BBT-1-0.2B: 200 million parameter financial model, including three variants trained with different pre-training methods, trained on 60 billion tokens:

  • (1) BBT-1-0.2B-001: 200 million parameters, financial model, T5 Decoder+Encoder architecture
  • (2) BBT-1-0.2B-002: 200 million parameters, financial model, T5+GPT
  • (3) BBT-1-0.2B-003: 200 million parameters, financial model, T5+UL2

• BBT-1-1B: 1 billion parameter financial model, T5 Encoder+Decoder architecture, pre-trained on 100 billion tokens of Chinese financial corpora including social media, financial news, securities research reports, and company announcements and earnings reports

• BBT-2-12B-Text: 12 billion parameter foundation model, GPT Decoder-Only architecture, without instruction fine-tuning, pre-trained on 200 billion tokens — performance has significant room for improvement, and developers can continue training or fine-tune for downstream tasks

• BBT-2.5-13B-Text: 13 billion parameter foundation model, GPT Decoder-Only architecture, without instruction fine-tuning, pre-trained on 200 billion Chinese and English tokens

Open-source download links

Models:

• Official website: https://bbt.ssymmetry.com/model.html
• GitHub: https://github.com/ssymmetry
• UCloud official platform — scan the QR code below

• Corpora: open-sourced nearly 100 billion tokens of pre-training data, including general and financial corpora. See: https://bbt.ssymmetry.com/data.html

• Evaluation datasets: open-sourced 8 Chinese financial large model evaluation datasets. See: https://bbt.ssymmetry.com/evaluation.html

• Paper: https://arxiv.org/abs/2302.09432

1. BBT-2-12B-Text General Model

BigBang Transformer [Qianyuan] is a large-scale pre-trained model based on the GPT Decoder-only architecture. Following the 2022 open-source release of BBT-1-0.2B, we are now officially open-sourcing the latest BBT series: BBT-1-1B, BBT-2-12B-Text, and BBT-2.5-13B-Text. The pre-training corpus covers 14 high-quality data sources spanning books, encyclopedias, papers, novels, news, policy documents, Chinese blogs, and social media. BBT-2-12B-Text was pre-trained on 70 billion Chinese tokens; the instruction fine-tuned base model can answer encyclopedic and everyday questions. BBT-2.5-13B-Text was pre-trained on 200 billion Chinese and English tokens. We are not currently releasing a Q&A dialogue interface for the base models. After open-sourcing, all developers can:

• Call the large model directly for dialogue
• Continue training on top of our checkpoint using their own corpora
• Fine-tune the large model for various downstream tasks

2. BBT-2-12B-Text+Code Code Model

Developers can test BBT's code Q&A capabilities on the SuperSymmetry website at https://www.ssymmetry.com (code generation scenarios only — it cannot answer questions unrelated to code). BBT-TC is the code model in SuperSymmetry Technology's recently released BBT-2 large model series. It was trained on a code dataset built on top of the 12-billion-parameter base model BBT-2-12B-Text, with its reasoning capabilities unlocked through supervised fine-tuning. In professional benchmarks, the model outperformed comparable models developed by other Chinese companies, ranking second only to GPT-3.5.

For more details, see the article "SuperSymmetry Technology's Code Large Model CodeBBT Ranks Near the Top in Professional Benchmarks, Second Only to GPT-3.5"

3. BBT-2-12B-TF-001 Financial Model

SuperSymmetry began developing applications for the financial investment sector as early as 2021, designing and training a large-scale parameter pre-trained language model called Big Bang Transformer (BBT). The company has released a Base version with 220 million parameters, a Large version with 1 billion parameters, and the latest BBT2, a 12-billion-parameter general-purpose model. The goal of BBT is to establish a unified AI algorithmic framework for financial investment — a transformer-based architecture capable of fusing and training on the different modalities of data involved in finance. By training large-scale parameter pre-trained models on this unified architecture, the SuperSymmetry team hopes to develop models approaching human-level intelligence in finance as parameters and training datasets continue to scale. As a foundational model for the financial sector, BBT provides fine-tuning services for downstream deep learning tasks across all financial investment, economic analysis, and business consulting scenarios.

The financial investment field has a vast number of institutions and practitioners. Large firms have the resources to hire algorithm engineers; smaller teams can't even afford basic text extraction algorithms. As algorithmic infrastructure for finance, BBT equips all practitioners with the same caliber of tools, putting the entire industry on the same starting line to compete for superior investment strategies — thereby driving more efficient information and factor flows in financial and economic markets.

(1) BBT-2-12B-TF-001 Financial Model has the following advantages:

Released a year before BloombergGPT, this Chinese financial large model has already secured paid contracts with numerous high-profile clients, including multi-billion-yuan quantitative funds in China and well-known Wall Street funds.

The most comprehensive financial dataset

To advance Chinese financial natural language processing, we have collected and scraped nearly all publicly available Chinese financial corpora:

• Financial and political news from all mainstream media platforms over the past 20 years
• All listed company announcements and financial reports
• Over ten million research reports from research institutes and consulting firms
• Millions of books on finance, economics, politics, and other social sciences
• User posts from financial social media platforms

(2) BBT-TF surpasses ChatGPT in announcement summarization tasks

Using identical inputs on the same announcement, we compared ChatGPT and BBT-TF summaries and found BBT-TF better suited to real-world applications. BBT-TF produces more concise summaries, precisely extracting key information from lengthy texts (ChatGPT above, BBT-TF below):

BBT-TF can also perform rounding calculations:

Given the financial industry's high precision requirements for numbers, BBT-TF accurately converts units:

BBT-TF can also interpret tabular information and generate corresponding text summaries:

ChatGPT fails to integrate financial real-world scenarios and overlooks certain key information:

We have developed 11 production-ready downstream tasks, available via API for professional financial developers, already generating paid revenue from domestic and overseas financial institutions:

Based on BBT-TF sentiment analysis downstream tasks, we have developed mature factor datasets. Example: single-factor分层回测 (layered backtest) on social media sentiment factors:

	IC	mean	t
1	0.04229	1.820765
50	0.038295	0.35027
std

category: alpha, same direction

returns (250 days)	volatility	sharpe (4% rf)	drawdown	win
37.08%	0.252574	1.309694	-8.82%	46.85%
19.24%	0.257211	0.592605	-8.52%	46.58%
3.33%	0.262807	-0.025426	-8.90%	46.52%
1.95%	0.269364	-0.076288	-9.41%	47.23%

Evaluation Dataset

To evaluate pre-trained models for the financial industry, the SuperSymmetry team released BBT-CFLEB — currently the most professional evaluation dataset for Chinese financial large models. It comprises eight standard language tasks to measure multi-dimensional capabilities across different models, establishing a cross-modal joint training architecture for text and time-series data based on Transformer, and promoting financial large model R&D. Now open-sourced on GitHub, with R&D details, evaluation data, model rankings, and downloadable open-source models for training downstream tasks available on the BBT model website. Teams are welcome to compete on the leaderboard.

4. BBT-2-12B-Image Text-to-Image Model

Built on the BBT-2 large language model, Supersymmetry developed BBT-Image, a text-to-image AIGC model. In partnership with Shanghai Huiyue Technology, a professional stock image company, they created the application platform ai.shenbi.pro for clients in the textile, printing, advertising, and gaming industries. In professional evaluations, BBT-Image has demonstrated significantly better performance than Stable Diffusion and other domestic large models for textile industry applications. BBT-Image can generate highly realistic images with controllable styles and aesthetics. In the textile industry, BBT-Image learns from images of different textile materials to generate fabric patterns with textures (seamlessly tileable), colors (brightness standardized for design purposes, unaffected by lighting variations in training samples), and design elements (freely combinable) — improving both efficiency and innovation in textile design. This technology can also be applied to home decor and other industries requiring pattern design.

5. BBT-2-12B-Science: Scientific Paper Model

• Large models will become powerful tools for discovering new scientific laws

Elon Musk believes the ultimate test for AGI is whether a model can discover new laws of physics. GPT-4 has scored well on medical licensing exams, bar exams, AP tests, and the GRE, but has yet to prove it can effectively discover or assist with knowledge that doesn't already exist in the human knowledge base. For Chinese large model R&D teams, targeting scientific discovery directly through large language models represents a possible path to surpass GPT-4. In scientific research, researchers can use language models to automatically extract and analyze topics, experimental methods, results, and conclusions from papers, thereby uncovering new scientific discoveries and research directions. BBT-Science is a large model for assisting scientific discovery, built by training the BBT model on tens of millions of research papers. Applied to research knowledge questions in physics, chemistry, biology, mathematics, and other disciplines, it offers three capabilities:

• Fast and precise knowledge retrieval. This capability is similar to large model conversational abilities in other domains.
• Novel ideas for frontier problems in a given research area. These ideas emerge from the model's massive data retrieval and recombination within that field, uncovering possibilities that previous researchers missed.
• Cross-disciplinary suggestions and insights drawn from multi-disciplinary training. This capability holds the greatest potential.

To evaluate scientific large models, Supersymmetry has partnered with Fudan University, Shanghai Jiao Tong University, Zhejiang University, Nanjing University of Aeronautics and Astronautics, Sun Yat-sen University, Beijing Normal University, and other institutions. They are calling on frontline researchers worldwide to jointly build ResearchQA, the largest evaluation dataset for scientific research questions. The dataset covers mainstream research fields including mathematics, physics, chemistry, biology, geology and geography, computer science, and electronic engineering. Scientific large models will become the foundational engine for global research capacity, accelerating scientific productivity. The dataset collects cutting-edge research topics directly from these fields as questions, with emphasis on evaluating the innovativeness of model responses. Researchers interested in contributing their research questions and answers to build this evaluation dataset can contact: researchqa@ssymmetry.com

Computer Science

Materials

Mechanical

Environment

Mathematics

Chemistry

Biology

Physics

• Applied Electronic Technology

• Materials Physics

• Materials Forming and Control Engineering

• Environmental Chemistry

• History of Mathematics

• Inorganic Chemistry

• Biological Science

• Classical Mechanics

• Electronic Science and Technology

• Materials Chemistry

• Industrial Design

• Environmental Biology

• Mathematical Logic and Foundations

• Analytical Chemistry

• Biotechnology

• Statistical Mechanics

• Computer Science and Technology

• Metallurgical Engineering

• Process Equipment and Control Engineering

• Environmental Toxicology

• Number Theory

• Organic Chemistry

• Bioinformatics

• Electromagnetism

• Microelectronics

• Polymer Materials and Engineering

• Industrial Design (Vehicle Engineering)

• Environmental Physics

• Dynamical Systems

• Physical Chemistry

• Ecology

• Theory of Relativity

• Computer-Aided Design and Manufacturing

• Composite Materials and Engineering

• Mechatronic Engineering

• Environmental Geoscience

• Functional Analysis

• Polymer Chemistry

• Integrated Science

• Quantum Mechanics

At the frontiers of mathematics, physics, chemistry, biology, geology, geography, and every other discipline, researchers battle against unsolved mysteries, continually expanding humanity's boundaries of knowledge. Having large models — compressed from learning hundreds of millions of papers and books — participate in the discussion and resolution of these problems will be among the most spectacular chapters in humanity's journey to understand the natural world. Consider the following frontier questions in cosmology:

6. Building a Mathematical Framework for Model Emergence

The research and development team at Supersymmetry Technologies is applying phase transition theory and renormalization group mechanisms from statistical mechanics to establish a mathematical framework for emergent phenomena in large models. Researchers at Google Brain compared the performance of different large models across various downstream tasks and found that model accuracy suddenly jumps from near zero when training reaches approximately 10^22 FLOPs. This phenomenon has now been observed across 137 distinct tasks. We define emergent capabilities as abilities that small models lack but large models possess. In a 2020 paper on scaling laws, the OpenAI team demonstrated power-law relationships between large model loss values and model parameter size, data volume, and compute.

In complex systems governed by physical laws, we find that power laws exist in critical states of second-order (continuous) phase transitions, giving rise to scale-free phenomena. Generally, observing a power law implies the presence of continuous phase transition behavior — superfluid and ferromagnetic transitions both exhibit this. Since 2003, experiments on biological neurons have shown that brain neuron firing also follows power laws and undergoes continuous phase transitions, suggesting the brain operates at criticality much like a sandpile that spontaneously collapses when it reaches a certain height. The renormalization group is an effective mathematical tool for describing phase transitions and criticality in condensed matter physics. Artificial neural networks are a highly abstracted model of biological neuron mechanisms. Since power laws have been discovered in large models based on artificial neural networks, we hypothesize that continuous phase transitions also occur during large model training, with some form of criticality present.

The renormalization group is an effective mathematical technique for analyzing phase transitions and criticality in condensed matter physics. It has successfully explained ferromagnetic transitions and superconductivity, with theoretical predictions closely matching experimental data. Researchers have already used the renormalization group to explain criticality in biological neurons. Therefore, Supersymmetry's R&D team is applying the renormalization group to build a mathematical framework for emergent behavior in large models, thereby connecting microscopic individual neurons to macroscopic 175-billion-parameter models through mathematical functions. By studying the underlying mechanisms of emergence through an effective mathematical framework, Supersymmetry aims to eventually achieve controlled emergence in large models.

7. Summary

• Supersymmetry Technologies is open-sourcing three large models:

1. BBT-1-0.2B: 200-million-parameter financial model, including three variants trained with different pretraining methods on 60 billion tokens:

   • BBT-1-0.2B-001: 200 million parameters, financial model, T5 decoder-encoder architecture
   • BBT-1-0.2B-002: 200 million parameters, financial model, T5+GPT
   • BBT-1-0.2B-003: 200 million parameters, financial model, T5+UL2

2. BBT-1-1B: 1-billion-parameter financial model, T5 encoder-decoder architecture, pretrained on 100 billion Chinese financial tokens including social media, financial news, broker research reports, and company announcements and earnings reports

3. BBT-2-12B-Text: 12-billion-parameter base model, GPT decoder-only architecture, without instruction fine-tuning, pretrained on 200 billion tokens. The model still has considerable room for improvement; developers can continue training on this general-purpose model or fine-tune it for downstream tasks

4. BBT-2.5-13B-Text: 13-billion-parameter base model, GPT decoder-only architecture, without instruction fine-tuning, pretrained on 200 billion Chinese and English tokens

• Supersymmetry Technologies has opened BBT-TC code model to the public. Trial link: https://www.ssymmetry.com/

• Supersymmetry Technologies calls on leading researchers worldwide to jointly build ResearchQA, the largest benchmark dataset for research questions.

Submission email: researchqa@ssymmetry.com

You can click "Read Original" to try the BigBangTransformer [Qianyuan] dialogue system.