Harnessing Collective Intelligence Instead of Creating Gods: This Company Takes a Unique AGI Path Different from OpenAI

Readers who watched the TV adaptation of The Three-Body Problem may recall a famous scene: The Sophon from Trisolaris locked down human technology and declared to Earthlings, "You are insects." But the ordinary, powerless Shi Qiang stood in a wheat field swarming with locusts and shouted: "The Trisolarans who see us as insects seem to have forgotten one fact that insects have never been truly defeated."

The Trisolarans saw the fragility of individual insects that you can easily crush an ant, swat a locust or kill a bee. But what they didn't see was that when these insects joined together, they could exert extraordinary power that was far greater than the sum of their parts.

Scientists have long recognised this phenomenon and named it "collective intelligence." This intelligence isn't controlled by a central brain, but emerges naturally through simple interactions and information sharing between individuals. It is a manifestation of collective wisdom and a wonderful and efficient way of collaboration in nature.

Actually, from a macro perspective, the continuous development and evolution of human society itself is a collective intelligence phenomenon. Most civilizational achievements are products gradually formed through long-term collective and socialized productive activities.

Can AI development follow this model? The answer is undoubtedly "yes". However, due to factors like low individual machine intelligence, "collective intelligence" has long struggled to emerge.

The advancement of generative AI may help solve these issues, bringing renewed attention to "collective intelligence."

"This wave of generative AI has elevated individual intelligence levels. With enhanced individual intelligence, collective intelligence can potentially achieve exponential growth," said RockAI CEO Liu Fanping in a recent interview with Synced.

Founded in June 2023, RockAI has independently developed China's first general large model with a non-attention mechanism Yan architecture. They've deployed this model across edge devices like smartphones, PCs, drones, and robots, while enabling "autonomous learning" capabilities on these devices.

All this serves a grand vision: making every device its own intelligence-systems capable of real-time learning and personalized autonomous evolution like humans. Liu believes that when these intelligent units with diverse capabilities and personalities collaborate, they can achieve data sharing, task allocation, and strategic coordination, manifesting more expansive and diverse collective intelligence. This would ultimately realize harmonious unity between personalization and collective intelligence, ushering in a new era of human-machine intelligence.

How will this be achieved? In the interview, Liu Fanping and Zou Jiasi (RockAI co-founder) shared their roadmap and latest progress with Synced.

A Unique AGI Path Different from OpenAI

As mentioned earlier, progress in "collective intelligence" research is constrained by individual intelligence levels, so the first challenge is making individual units sufficiently intelligent.

In terms of "intelligence," OpenAI's models are outstanding. However, their current approach focuses on training single super-intelligent large models. This path faces challenges due to its heavy reliance on massive data and computational resources, creating sustainability issues in energy, data, and costs.

Moreover, relying on a single super-intelligent model for all tasks represents a highly centralized approach, often encountering intelligence growth bottlenecks in practice. Single models lack flexible adaptability and collaborative effects, limiting their intelligence improvement speed.

Could OpenAI adopt collective intelligence in the future? The answer remains unclear. However, we can see that current Transformer architectures used by OpenAI and most companies present obstacles for building collective intelligence units.

The first obstacle is high computational demands. The Attention-based Transformer architecture requires substantial computing resources, with computational complexity at O(n²) (n is sequence length). This means costs escalate rapidly with longer input sequences. Building collective intelligence requires multiple unit models collaborating on low-power devices (e.g., drones, smartphones, robots). Without quantization or pruning, Transformer models struggle with direct deployment on such devices. Therefore, we can see many companies resort to these techniques for edge deployment.

But for collective intelligence, merely getting the models to run is not enough; they must also have the ability of autonomous learning. In Liu Fanping's view, this is crucial.

He elaborated that in a swarm without autonomous learning, the smartest individual dominates others' decisions, making them can only follow its lead. In this case, the upper limit of collective intelligence is the level of the smartest individual and cannot be surpassed. But with autonomous learning, each agent can independently improve, approaching the smartest individual's level. Moreover, it facilitates knowledge sharing, similar to human knowledge transmissionThe most fatal problem of quantization and pruning is that they undermine models' autonomous learning capacity.. This elevates all agents' intelligence, enabling exponential collective growth far beyond simple aggregation.

The most fatal problem of quantization and pruning is that they undermine models' autonomous learning capacity."After compression, quantization and pruning, models lose relearning ability because their weights become irreversibly altered. It's like hammering a screw into a wall, if the screw is damaged during the hammering process, it becomes very difficult to take it out and reuse it, and it becomes impossible to make it sharper," Liu explained.

Thus, the path to realizing collective intelligence becomes quite clear:

● First, you need to change the architecture and innovate a new one to overcome Transformer's limitations.

● Second, deploy models based on this architecture across edge devices for optimal adaptation.

● Third, a more important point is that these models must achieve autonomous learning and evolution on edge devices.

● Finally, the agents formed by combining these models with edge devices must autonomously collaborate to accomplish tasks.

Each of these phases is not simple:

● In Phase 1, the new architecture must not only be low-computing and deployed natively lossless on the end-side, but also have performance comparable to the Transformer architecture, which ensures that a single individual is smart enough to learn on its own.

● In Phase 2, high compatibility of the "brain and body" involves different modalities at the perception level and in data processing. Each device has different requirements, which increases the complexity of model and device compatibility.

● In Phase 3, making the model learn even after deployment means challenging the existing mechanism of complete separation of training and inference, so that the model parameters can be adjusted on the end-side as well, and that the adjustment is fast enough and cheap enough. This involves challenging the traditional backpropagation mechanism and requires very bottom-up innovation.

● In Phase 4, the main challenge is to achieve effective collaboration between agents. This process requires the agents to discover and form the best solution to accomplish a task autonomously, rather than relying on a human-set or programmed pre-determined solution. Agents need to decide on the way to collaborate based on their level of intelligence.

These challenges dictate RockAI's unique path, which is different from OpenAI and challenges conventional "consensus" methods.

Liu noted they've achieved milestones in Phases 1-2, with experimental concepts for Phases 3-4.

The Unit Large Model for Collective Intelligence-Yan 1.3

Phase 1's milestone is a large model using the Yan architecture (not Transformer or its variants). Version 1.0, released in January, is a general language model without Attention mechanisms. Reportedly, it achieves 7x training efficiency, 5x inference throughput, and 3x memory capacity compared to same-parameter Transformers. Moreover, it supports complete private deployment without pruning or compression, running natively on mainstream consumer CPUs and edge devices.

After over six months of efforts, its latest version called Yan 1.3 has just been released.

Yan 1.3 is a 3B-parameter multimodal model capable of processing text, audio, visual inputs, and outputting text and speech, enabling human-like multimodal interaction.

Despite the small number of parameters, it outperforms Llama 3 8B. It also uses less training data and computational resources than Llama 3. This is a leading achievement among non-transformer models with lower training and inference costs, making it more viable for industrialization and commercialization.

These capabilities stem from efficient architectural design and algorithmic innovation.

Architecturally, RockAI replaced Transformer's Attention with an MCSD (multi-channel slope and decay) module while preserving token relationships. MCSD emphasizes efficient information transfer, ensuring only critical data progresses with O(n) complexity, boosting overall efficiency. It excels in validating feature effectiveness and token correlations.

Algorithmically, RockAI proposed a brain-like activation mechanism. It's a zoned activation approach where Yan 1.3 adaptively engages specific neurons based on learning types and knowledge domains rather than having the full amount of parameters involved in training, akin to how driving and writing activate different brain regions. This applies to inference, with neuron selection governed by bio-neuronal algorithms.

At GTC this year, Illia Polosukhin (co-author of the Transformer paper) noted that simple tasks like "2+2" may engage trillions of parameters in large models. He argued adaptive computation is essential for it determines appropriate resource allocation per task. RockAI's brain-like mechanism offers one implementation.

This may resemble Mixture of Experts (MoE), but Liu clarified fundamental differences: MoE uses fixed "expert" networks voting on task allocation, yielding predictable outcomes. The brain-like mechanism lacks "experts" or voting-it dynamically selects neurons, each neuron is valuable, and the process of selection is a self-learning process.

This zoned activation further reduces Yan architecture's training and inference complexity and computational load beyond MCSD's benefits.

"This also mirrors human brain operation. The brain consumes just about 20W. If all 86 billion neurons were involved in computing every time, the bioelectric signals produced by the brain would be insufficient," Liu noted. The mechanism has received theoretical and experimental validation from neuroscientists, with patents filed.

Toward Collective Intelligence with Edge Devices

At Yan 1.3's launch, we saw deployments on PCs, smartphones, robots, and drones. Given that Yan 1.2 even ran on Raspberry Pi, this edge deployment progress isn't surprising.

So why insist on edge deployment? Can't cloud models suffice? Zou Jiasi explained that models require tight integration with hardware. For robots, many device parameters can't sync with cloud models. Edge models enable better motor coordination and cooperation between cerebrum and cerebellum.

Besides, we know that these edge agents' potential is just emerging. After all, the innovative goal above isn't merely edge execution (currently many models achieve this), but autonomous learning as "unit large models for collective intelligence" that continuously evolve. No matter Yan architecture's "zero-compression, zero-pruning" deployment or zoned activation all serve this purpose. It's a key difference between RockAI and other edge-AI companies.

"Comparing a 10-year-old to a 30-year-old PhD, the latter definitely has broader knowledge. But we can't assume the child won't surpass the PhD later. With strong self-learning ability, the child may progress faster than the PhD in the future. Thus, we believe that autonomous learning is the true metric of model intelligence,"Liu said. This capability represents RockAI's "scaling law."

To achieve this, RockAI proposed "training-inference synchronization", which allows models to update knowledge effectively and continuously in real-time during inference, building unique knowledge systems. This mirrors humans speaking while listening and internalizing knowledge, demanding advanced underlying tech.

For this reason, RockAI's team is seeking a better solution for backpropagation. Some prototypes of the method have already been developed and presented at WAIC. However, their prototypes still face some challenges, such as latency. In the subsequent release of Yan 2.0, we are expected to see the demonstration of the upgraded prototype version.

Then, how will intelligent devices interconnect to manifest collective intelligence? Liu shared his preliminary ideas:

First, they'll form a decentralized dynamic system where each device autonomously learns and decides without relying on central control. Meanwhile, they'll share local data or experiences and exchange information with each other via high-speed networks. Thus, cooperation can be initiated when needed to leverage collective knowledge and resources for efficient task completion.

A "Niche" Path with Challenges and Opportunities

Within China's AI landscape, RockAI's approach is quite "niche" due to its foundational innovations. In Silicon Valley, there are quite a few people doing similar underlying research, and even Hinton, the ‘father of neural networks’, has expressed concern about some of the limitations of backpropagation, particularly its incompatibility with the biological mechanisms of the brain. However, no one has found a particularly effective method yet, so there is no obvious technology generation gap in this direction. For domestic companies like RockAI, this is both a challenge and an opportunity.

Liu believes collective intelligence is a viable path toward broader AGI, given its solid theoretical foundation: "Without collective intelligence, human civilization and technological progress wouldn't exist." With the successful development of autonomous architecture model and the construction of a diversified hardware ecosystem, we believe we are gradually approaching this goal.

References:

http://news.sciencenet.cn/sbhtmlnews/2023/2/373354.shtm

http://lib.ia.ac.cn/news/newsdetail/68393

http://www.shxwcb.com/1205619.html

http://mp.weixin.qq.com/s/t6TurjgHHxmC2D--c9-fcg