Since 2021, the metaverse has been gaining more attention from investors worldwide. By integrating them into business processes, new technologies, including digital humans, have the potential to replace and outperform human beings in several areas. In turn, this may help enterprises reduce costs and increase efficiency.

In this article, we invited Mr. Gao Ruisheng, Product Director for Digital Entertainment at SenseTime, to share his perspective on capital investments, market implementation, core technologies of the metaverse, as well as typical constructions and applications of digital humans.

The role of the AI engine layer

Creating a metaverse, or a virtual parallel world, may contain three major elements: avatars, AI digital humans (software agents), and 3D Space (3D reconstruction of the physical world).

As PC traffic shifts to mobiles, the untapped traffic is declining, which leads us to finding new opportunities. Furthermore, the mechanics of transferring information have been changing all the time, from text and voice to images and videos, and finally to the level of three-dimensional holograms or a reality with five senses through a brain-computer interface.

Metaverse, in which information is three-dimensional too, is on its way to becoming increasingly crucial for the whole technology industry.

As shown in the above image, there are five layers in the metaverse system: the infrastructure; displaying hardware; OS; 3D engine; and applications.

Nowadays, customers are interested in developing non-traditional applications of the metaverse, but the production of professional content is a challenge. The solution to this problem would require the aid of an AI engine, which is SenseTime's primary objective.

To understand this AI engine, we have to learn about the relationship between the human brain and artificial intelligence. This AI engine has cognitive abilities, including the cognition of humans and scenes. Meanwhile, it could generate content, which is similar to dreaming, where we invent characters and scenes in our minds.

Accordingly, the AI engine layer enables a digital human to have a strong engine with three significant capabilities.

For SenseTime, the first thing we want to do is make it easy for people to get a digital human. Using the capability of image generation, you can quickly generate thousands of avatars, which can be ACG, 3D hyper-realistic, or Korean anime styles.

Meanwhile, AI algorithms can be used to create mature NPC groups, such as AI digital humans, which possess three fundamental characteristics:

First, it is human-like in appearance. In addition, it displays facial expressions, gestures, movements, and behaviors. Second, it has a human-like brain based on our multimodal natural language processing (NLP) technology. Using the information the external environment provides, the NLP brain can interpret, communicate, and provide user services according to their needs. With this technology, cities and enterprises can reduce human resource costs and accelerate the digital transformation.

Additionally, the 3D HD reconstruction technology provides an efficient method for creating an exciting digital environment.

In light of what has been discussed above, we can construct intelligent solutions for the digital world and create virtual spaces for various scenes in a city. With this virtual environment, one can enjoy multiple activities with their families and friends and experience face-to-face immersive communication, exchange, and experience across physical distances.

In virtual content generation, the first step is the rapid generation of avatars. A city that wishes to attract young people may enable them to take a selfie and generate an exclusive cartoon figure with a single click. Through avatars, people can embark on a new journey of reality and imagination.

The second feature is the instant creation of real-world 3D high-precision reconstructions. It is possible to fly a drone and quickly reconstruct a high-precision 3D scene. Additionally, a collaborative team can rebuild a high-precision 3D space that is more creative and accurate.

By utilizing the previously created avatar, you may be able to see the landscape of the city.

Companies such as Meta have made such an attempt, and one example would be their Oculus Horizon Worlds, in which a reconstruction of Los Angeles and San Francisco has been built, enabling users to view the landmarks from anywhere in the world.

Plus, virtual government, business offices, and virtual exhibitions can also be constructed with this technology. Business operators can develop smart digital humans, providing convenient services like hospitality and consulting services, so that users can enjoy convenience even from their homes. An exhibition requires a panoramic display, not simply video conferencing software, and Oculus already has applications for big screens. 

The core technologies of digital humans

In the future, the idea of a digital human will become familiar to people as "super agents" (SenseMARS Metahuman). A digital human can display human characteristics like facial appearance, gestures, and a biological brain. Therefore, it can replace part of the workforce and even exceed what humans are capable of. With intelligent assistants, customer service agents, content presenters, and brand ambassadors, enterprises can reduce human resource costs, increase productivity, and support cities in their digital transformation.

Digital human platform features can be classified into three main categories:

First, the knowledge systems of digital humans can be managed in a unified management platform, for example, by clarifying what the digital human says, how the Q & A should be conducted, or what services should be rendered. Furthermore, the image of the digital human can be customized via remote control or OTA upgrade channels.

Second, we can employ algorithmic modeling within our respective platforms to render and motivate digital humans.

Moreover, the application layer is expanded to enable digital humans to be deployed in real estate, superstores, parks, hotels, and multiple office buildings while they can be viewed and interacted with using mobiles, PCs, tablets, large screens, AR/VR glasses, in-car infotainments, all-in-one terminals, etc.

SenseTime has implemented several product forms and functions, including online, offline, C-end scenes, etc. The company has five significant advantages for digital humans:

Multiple avatars for selection;

Quick generation of virtual humans;

Elaborate representation;

Various driving channels;

Industry-leading AI algorithms (e.g., the algorithm model of self-research STA that improves mouth shape accuracy; the self-research NLP system that enhances Q & A capabilities).

Application scenarios for digital humans

An example of a digital human application is virtual brand ambassadors.

Since last year, celebrities' scandals and moral issues have bothered many PR managers. As A-listers also have busy schedules, it is natural for companies looking for quality brand ambassadors to find some innovative solutions. 

For the traditional production of a film and television-grade CG avatar, costs are pretty high, and the cycle is rather lengthy. SenseTime enables a more accessible approach to human generation by utilizing a fast and efficient system. This will minimize the costs associated with multi-media resources.

Next come online and offline financial services. Online businesses can integrate the digital human into their apps, H5 pages, and applets to provide intelligent customer services and financial recommendations. Through dialogue interaction, digital humans can guide clients to use apps and recommend financial products to them.

The offline scenario is more straightforward. Using digital humans can reduce the cost of human resources, substitute partially for agents, and serve as a resource for cost reduction and improved efficiency.

New media is another area where innovation is taking place. As virtual live, ACG, and short videos have gained traction in recent years, they are no longer abstract symbols but represent a real economy with countless hard-core fans. Similarly, all of these emerging markets are technology and content-driven, and SenseTime has already adopted proactive approaches to develop its capabilities and products concerning underlying algorithms, SDKs, and platform deliveries even before all these techniques become mature.

SenseTime offers comprehensive technologies explicitly tailored to the customer's needs, including virtual beauty makeup features, avatars for short videos and ACG applications, and platforms for generating digital human videos. By doing so, traditional industries can quickly adapt to this changing market and remain competitive.

Digital humans also play an important role in cultural tourism. A digital-human machine from SenseTime has been deployed for museums, which can provide information such as the locations of souvenir shops, exhibition areas, and restrooms to visitors. Additionally, it has been thoroughly trained and familiarized with the contents of the history and, therefore, can give answers to any questions visitors may have. On top of that, there is a digital human on the super screen, which can display information, assist staff, greet visitors, etc.

SenseTime's digital humans have an extensive track record of success in new retail and virtual hosting. In addition to having the ability to take on all of the general grocery guiding duties, SenseTime's intelligent agent can also become the Ms. Know-All of the supermarket due to its AI technology. Her expertise can assist customers, from finding parking lots to offering discounts on products. It can host various online and offline events and adapt its appearance, style, and even language according to the event's theme.

To put it simply, digital humans will provide solutions for a wide variety of industrial sectors. The advancement of other technologies may hold the key to the creation of more digital humans in the year 2022, as well as the development of the metaverse.