Supersymmetry Technology Releases BigBang Transformer [Qianyuan]: A 1 Billion Parameter Financial Large-Scale Pre-trained Language Model That Can Read Numbers Supersymmetry Technology has released the BigBang Transformer [Qianyuan], a 1 billion parameter financial large-scale pre-trained language model designed to enable pre-trained language models to understand numerical data.

Lead: Linear Capital portfolio company Symmetry Technologies releases BigBang Transformer [Qianyuan], a 1-billion-parameter financial pre-trained language model. The BBT large model is built on a time-series–text cross-modal architecture, jointly training on both textual and temporal data. It achieves nearly 10% higher accuracy on downstream tasks compared to same-scale T5 models, and substantially improves R² scores for time-series forecasting. The cross-modal design enables the language model to detect shifts in temporal data and articulate its findings in human language. BBT can be applied to factor mining for quantitative investment strategies, multi-factor portfolio construction, and broader time-series data analysis for data visualization and IoT applications. The model's ultimate goal is to achieve human-level analytical capability in a pre-trained foundation model, constructing a general AI architecture that can be deployed across industries.

01 The Limitations of General-Purpose Large Models

OpenAI's GPT-3, Google's LaMDA and PaLM, and other 100-billion-plus parameter language and multimodal models approach or surpass human-level performance in writing, text-to-image generation, and dialogue. But these models share several common weaknesses:

① They are pre-trained on general-purpose corpora and perform well in broad scenarios, yet show clear deficiencies in specialized domains. That's why GPT-3, Wudao, Pangu, and similar models are typically demonstrated through fiction continuation, poetry composition, or human conversation. When it comes to serious professional applications, there's been plenty of thunder but little rain. No large model-based product has yet achieved scaled industry deployment, and the reasons behind this merit deeper investigation. Where exactly is the capability boundary of models trained solely on general corpora without domain-specific data? If the Symmetry team can demonstrate that models trained on industry data achieve better accuracy, does this suggest that the overall design of existing large models needs fundamental revision to achieve cross-industry generalizability?

② While pre-trained multimodal models like DALL-E 2 have produced stunning text-to-image results, progress has been limited on more practical and complex modalities such as time-series and tabular document data — modalities that dominate real-world work scenarios. Beyond processing the three common modalities of language, speech, and image, the ability to read and analyze data represents a distinctive human intelligence, one where humans can process language and data in parallel to reach conclusions. Can large models replicate this human capacity for data analysis, thereby enabling genuinely widespread industrial deployment?

Symmetry Technologies specializes in developing algorithms and data products for finance, media, manufacturing, and other industries. The company has designed and trained BigBang Transformer Qianyuan (BBT), a large-scale pre-trained language model targeted at financial investment applications, releasing a Base version with 220 million parameters and a Large version with 1 billion parameters. The team has also open-sourced BBT-FinCUGE, a comprehensive evaluation dataset for financial pre-trained models, on GitHub. BBT adopts the T5 encoder-decoder architecture to handle both NLU and NLG downstream tasks. The Symmetry team assembled a financial industry dataset and built a Transformer-based architecture for cross-modal joint training of text and time-series data.

Large models represent one path toward Artificial General Intelligence (AGI). Symmetry Technologies believes that data analysis capability is one of the foundations for achieving AGI. The company has partnered with Fudan University School of Computer Science's Xiao Yanghua Knowledge Works Lab, Zhejiang University's Xu Renjun Lab, and faculty from Nankai University and Beijing Normal University's School of Artificial Intelligence to advance foundational AGI algorithm research across theory, architecture, and algorithm implementation — building the infrastructure for AGI industrial applications. This research receives computing infrastructure support from Gansu Gaotai's "East Data, West Computing" initiative.

Using Google's T5 framework as a baseline, BBT's experiments validate the following conclusions:

• Domain-specific pre-training on professional datasets improves average downstream task accuracy by nearly 10% compared to same-scale T5 models.
• The proportional mix of downstream task corpora affects downstream accuracy.
• Source Prompt-based prompt learning tailored to downstream task categories substantially improves accuracy.
• BBT's time-series model achieves significant R² score improvements over standard Transformers for multivariate time-series forecasting.
• Joint training on text and time-series data enables the model to interpret what numerical changes mean in the real world.

02 A Pre-trained Model Architecture Focused on Joint Time-Series–Text Training

Traditional time-series models typically rely solely on temporal information itself, neglecting the external dependencies of time-series data. For instance, fluctuations in stock prices or economic indicators at a given moment are not fully determined by preceding data alone. Language models possess powerful capabilities for representing textual information; combining them with time-series models allows world knowledge to support time-series tasks in textual form, while information embedded in temporal data can in turn strengthen the language model's comprehension.

To this end, the Symmetry team designed a Transformer-based time-series–text cross-modal pre-trained model — among the earliest industry architectures dedicated to joint training of these two modalities. The pre-training objective predicts time-series data at moment T using textual and temporal information prior to T. Both time-series and text/image data are fed simultaneously as embeddings into a bidirectional Transformer encoder; output vectors then pass to a decoder with three branches: NLU, NLG, and Time Series.

BBT designed a universal module for vectorizing time as embedding inputs. Multivariate time series are influenced by signal impulses across both spatial and temporal dimensions, with activation ranges forming a continuous spectrum that can be analyzed across four categories: low-frequency local impulses, low-frequency global impulses, high-frequency local impulses, and high-frequency global impulses. Here, "low-frequency"/"high-frequency" describes activation range from a temporal perspective, while "global"/"local" describes it from a spatial perspective.

• "Low-frequency" means the impulse changes gradually, tending to remain stable over longer periods;
• "High-frequency" means the impulse changes dramatically;
• "Global" means the impulse affects all time series similarly;
• "Local" means the impulse affects only individual time series, or applies differently across series.

Based on this, Symmetry proposes DWT-ST2Vec, a universal, model-agnostic, learnable vector time representation component applicable to diverse architectures and downstream tasks. This component decomposes high- and low-frequency components from both spatial and temporal dimensions, enabling more comprehensive sequence learning.

03 The Most Comprehensive, Largest-Scale Financial Investment Dataset in Academia and Industry

The quality, quantity, and diversity of a corpus directly impact language model pre-training effectiveness. Existing Chinese financial pre-trained language models, such as FinBERT and NVIDIA's FinMegatron, have been limited in both the scale and diversity of their pre-training corpora.

To advance Chinese financial NLP, Symmetry gathered and crawled nearly all publicly available and otherwise obtainable Chinese financial textual data: financial, political, and economic news from all major media platforms over the past 20 years; announcements and financial reports from all listed companies; millions of historical research reports from research institutes and consulting firms; millions of books on finance, economics, politics, and social sciences; announcements and documents from over 40 central government ministries and local government websites; and social media user posts. From this, the team cleaned and organized BBTCorpus, a large-scale Chinese financial corpus spanning five categories totaling over 300 GB and 80 billion tokens of high-quality, diverse data — currently the most comprehensive and largest financial investment dataset available. The scale distribution is shown in Table 1.

Table 1: BBTCorpus size distribution. Note that listed company announcements and research reports are in original PDF format.

04 Innovative Pre-training Methods that Substantially Improve Language Model Accuracy: Similarity Sampling and Source Prompt

To validate the effectiveness of domain-specific pre-training, the Symmetry team compared their model against t5-v1_1-base-chinese-cluecorpussmall, pre-trained on the general corpus CLUECorpus-small. Results are shown in Table 2.

The Symmetry team made innovative modifications to T5's pre-training approach for their specific challenges.

First, they proposed a corpus source similarity-weighted sampling algorithm to address corpus sampling issues. Given the enormous size of their corpus — only about 10% of which could be sampled during the full pre-training process — the model necessarily had to randomly sample from different sources. Simple random sampling would effectively mix sources by size, yielding a pre-training subset with announcement:research report:news:stock forum:Xueqiu ratios of roughly 105:11:30:74:44. The team proposed that similarity-weighted sampling relative to evaluation benchmark texts, rather than simple random sampling, is a more rational approach. Models trained on the similarity-weighted subset achieved average improvements of 0.7% on evaluation benchmarks; results are shown in Table 2.

This innovation applies not only to financial domain pre-training but can extend to other fields with multiple heterogeneous corpus sources, such as biomedicine and law. Building on this, the team further scaled the model to the 1-billion-parameter Large tier; results are also shown in Table 2.

Table 2: Scores represent average performance on evaluation benchmarks. T5-base denotes t5-v1_1-base-chinese-cluecorpussmall. "ss" indicates the team's first innovation, Similarity-weighted Sampling of corpus source. Base models have 220 million parameters; Large models have 1 billion.

The team further pioneered Source Prompt (SP) to address heterogeneous corpus mixing — placing a token indicating source origin before each corpus during pre-training. For the corpus: "According to the National Bureau of Statistics, in May 2022, national consumer prices rose 2.1% year-on-year," pre-training prepends the source prompt: 【News】, yielding: "【News】According to the National Bureau of Statistics, in May 2022, national consumer prices rose 2.1% year-on-year," followed by standard MLM pre-training. Source Prompt improved upon the Similarity Sampling model by 3.21% on the Base model.

Table 3: Performance of T5-base and various BBT models across 8 downstream tasks.

05 Universal Time Vector Representation Component DWT-ST2Vec Connects Different Models

BBT's fundamental capabilities for processing time-series data include:

• Providing DWT-ST2Vec, a universal, model-agnostic, learnable vector time representation component that enables time to be embedded as input to the encoder for joint learning with text.
• Achieving more accurate multivariate time-series forecasting.
• Decomposing time-series data along "global-local," "period-trend," and "low-frequency-high-frequency" dimensions.
• Through joint learning with text, enabling the large model to generate natural language descriptions of time-series changes.

For evaluation, 40 randomly selected domestic listed companies were used, with opening stock price time series as the primary target. Training used 4,000-length sequences since IPO; testing used sequences 4,000–4,200. Evaluation combined MSE, RMSE, MAE, and MAPE metrics. Compared to a Transformer baseline, trained models achieved average improvements of 0.5%–2% across MSE, RMSE, MAE, and MAPE.

BBT's time-series–text cross-modal architecture can identify stock price movements and trigger NLU capabilities to generate commentary resembling that of analysts and retail investors. Input stock prices:

The model can draw on its learning from massive news corpora to write commentary resembling professional financial journalists, such as:

It can also discuss market trends like retail investors:

BBT's time-series–text cross-modal architecture enables the model to read company financial reports and news to generate development trend analyses; to learn years of e-commerce sales data and product characteristics to forecast future sales and produce targeted marketing reports; or to learn manufacturing equipment monitoring data to generate maintenance fault reports comprehensible to non-specialists.

06 BBT-KG: Dynamic Causal-Tracing Event Knowledge Graph

The Symmetry team constructed a knowledge graph covering 200,000 Chinese primary-market companies and 4,500 A-share listed companies for knowledge-enhanced language model learning. BBT-KG differs from existing financial knowledge graphs in that the team leveraged language model capabilities to build dynamic associations between news events and enterprises, and causal relationships between events themselves. This enables the model to assess the impact of newly occurring events on companies and markets, and to trace market fluctuations to their root causes.

07 Applying BBT to Build Novel Quantitative Investment Factors

BBT Empowers Multi-Factor Strategy Development

The Symmetry team applied BBT to compute individual stock sentiment indices, monitoring adjacent-period sentiment changes and selecting prominent shifts as long-short factors for quantitative strategy construction, ultimately generating returns far exceeding the market. Backtesting revealed the sentiment index's outstanding stock selection capability, demonstrating that the model effectively learns financial and economic texts and quantitatively reflects market information, creatively providing alternative factors. Beyond market sentiment calculation, BBT's multidimensional capabilities have broad financial applications.

For example, using BBT's event extraction capability, similar events or news can be extracted for comparison with price-volume data to study how quickly different events transmit to markets. BBT can also learn interdependencies among economic agents in supply chains through the team's proprietary financial knowledge graph, using machine learning to eliminate factor collinearity — bringing disruptive innovation to traditional linear regression multi-factor models.

Additionally, BBT's negative news identification capability can add real-time public opinion monitoring to credit risk assessment systems, while its news classification capability helps financial analysts rapidly process large information volumes for more comprehensive, objective conclusions.

08 Benchmark Evaluation Dataset: The First Chinese Financial NLP Evaluation Benchmark

Evaluation benchmarks play a crucial guiding role in NLP development. Yet as Chinese financial NLP research and applications flourish, the industry lacks an authoritative benchmark. To address this, the Symmetry team proposed BBT-FinCUGE, open-sourced at:

GitHub.com/ssymmetry/BBT-FinCUGE-Application

This is a Chinese financial natural language understanding and generation evaluation benchmark with the following characteristics:

① Professionalism: All dataset filtering and annotation involved financial experts.

② Practicality: All tasks were scored for practical utility by financial experts, serving as the basis for task selection and final scoring.

The benchmark comprises eight datasets:

• Forum Sentiment Analysis (FinFE)

On retail investor forums such as Guba and Xueqiu, users generate massive comment volumes daily, containing both emotional expression and rational price predictions. This dataset requires models to learn and predict sentiment indices (0, 1, 2 representing negative, neutral, and positive respectively).

• Event Extraction (FinQA)

Event extraction automatically identifies event occurrences from text, extracts event arguments, and structures the data — including detection and argument extraction for financing, IPO, acquisition, and other corporate events. (For better cross-model comparison, the team reformatted this as reading comprehension QA.)

• Causal Event Extraction (FinCQA)

Unlike standard event extraction, causal event extraction focuses on identifying pairs of causally related events and their arguments from text, structuring them as data. The team's causal event dataset covers commodity domain causal event identification, including typhoons/earthquakes, supply increases/decreases, demand increases/decreases, price rises/falls as potential cause and result events, with corresponding products, regions, and other parameters. (Reformatted as reading comprehension QA for cross-model comparison.)

• News Text Summarization (FinNA)

Chinese financial news summarization. Sourced from Sina Finance's large-scale Chinese short news corpus, containing 20,000 real Chinese short texts with corresponding summaries.

• Relation Extraction (FinRE)

A manually fine-annotated finance domain dataset. Given a sentence and head-tail entities, the model predicts their relationship. Annotated from Sina Finance news, with named entities as commercial companies and 44 financial domain relation categories (bidirectional), including ownership, shareholding, competition, acquisition, transaction, cooperation, reduction, and other finance-specific relations.

• Negative News Identification and Subject Determination (FinNSP)

This dataset contains two tasks: Negative information determination: Whether the text contains negative information about a financial entity. If the text contains no negative information, or contains negative information not involving financial entities, the result is 0. Negative subject determination: If task 1 identifies financial entity-related negative information, determine which entities in the entity list are the subjects of that negative information.

• News Classification (FinNL)

Classify financial news into one or more categories related to its content. Sourced from Sina Finance, with 14 categories: company (individual stocks), industry (sectors), broad market, China, international, economy, policy, futures, bonds, real estate, foreign exchange, virtual currency, COVID-19, and energy.

• Event Subject Extraction

The primary goal is to extract subjects of specific event types from real news corpora. Given text T and event type S, extract the event subject of type S from T. Input: a text passage, event type S; Output: event subject.

09 Developer Services: Open APIs for Financial and Non-Financial Industry Developers

Building the BBT Large Model Developer Ecosystem

The Symmetry team is opening 11 API capabilities to developers across financial and non-financial industries to build the BBT large model developer ecosystem. The first batch includes: knowledge graph, article summarization, social media sentiment identification, news sentiment identification, news classification and tagging, named entity recognition, relation extraction, event extraction, causal event extraction, announcement extraction, and negative news and subject identification.

API documentation: https://www.ssymmetry.com/newproduct/bbtlink

10 Foundation Model for Finance and Economics

BBT 1.0 aims to establish a unified AI algorithm framework for financial investment, building a Transformer-based architecture capable of jointly training on diverse modalities involved in financial investment. On this unified architecture, large-scale parameter pre-trained models are trained; as model parameters and training datasets continue to grow, the Symmetry team hopes to develop models approaching human-level intelligence in the financial domain.

As a foundation model for finance, BBT provides fine-tuning services for all deep learning downstream tasks in financial investment, economic analysis, business consulting, and related scenarios. The financial investment field has numerous practitioners and institutions: large firms can afford to hire algorithm engineers, while smaller teams cannot afford even basic text extraction algorithms. BBT serves as algorithmic infrastructure for the financial domain, equipping all practitioners with comparable tools and placing the entire industry on the same starting line to compete for superior investment strategies — thereby promoting more efficient information and factor flows in financial and economic markets.

Teaching models to read numbers is a capability BBT specifically develops through its time-series–text cross-modal architecture — one of the most core capabilities in the pursuit of AGI. The ability to identify patterns and regularities in massive time-series data and accurately map them to the real world through pre-trained language models, building a bridge between the data world and human language, will revolutionize broader digital technologies including business data analysis, data visualization, and database technology. BBT's applications extend beyond finance to manufacturing, IoT, smart cities, and large-scale internet data analysis — wherever time-series data processing is central.