Nvidia’s eye contact effect is a game-changer for video creators

Video conferencing, podcasts, and webinars surged in popularity during the pandemic years of 2020 and 2021, with remote working becoming part of the new normal. With the pandemic now in the rearview mirror, video communication techniques showed no signs of slowing down.

What amused me was that despite the ubiquity of video communications, the way we often appear unflattering on camera using low resolution, underpowered webcams gets too little attention. Poor lighting, especially when using video calls from home, is undoubtedly a big problem. The sub-HD resolution webcams built into most laptops, even high-end ones, don’t help.

Without the professional assets available in a professional television studio, politicians, celebrities, and industry pundits often look horrible when interviewed remotely from their homes.

Routine video conferencing calls from home are particularly vulnerable to the appearance of “amateur hour”, especially during a formal presentation where the wandering gaze (e.g. not looking directly into the webcam) can distract the viewer.

The location of the webcam is responsible for this undesirable effect because the camera is usually integrated into the top of the laptop panel or on a separate stand that is difficult to place in front of a desktop screen.

Since the typical videoconference using a desktop or laptop computer lacks the proper teleprompter functionality, which is complex, cumbersome and expensive, it is nearly impossible to read speaker notes without avoiding the annoying phenomenon of a horrible webcam angle looking up or down. your nose.

Are there any quick ways to fix the gaze problem?

There are several ways to alleviate this problem in a typical home desktop or laptop computer setup. However, these approaches are strictly trick and do not eliminate the problem.

A few companies provide tiny external webcams, often with a built-in microphone, to reduce device size and allow placement in the center of your screen, in front of any text or the video application’s viewing window itself. that you use.

These cameras use a thin wire that is draped and clipped to the top of the screen. This way you are looking directly into the webcam and can see most, but not all, of the presentation or text you are presenting.

Yet another method is to use a piece of clear acrylic plastic which allows you to mount almost any webcam and hook it to the top of the screen so that the webcam hangs in front of the center point of the screen .

The advantage of this approach is that it allows you to use your favorite webcam. The downside is that the webcam’s size and acrylic plastic device often obscures a good portion of the screen, making it less useful as a teleprompter alternative.

Later, we can see laptop and PC screens with built-in webcams behind the LCD panel, which are invisible to the user. While this is an ideal solution to the problem I described above, the downside is that the cost of these specialized displays will be very high, which most manufacturers will be reluctant to offer due to implications for price elasticity.

AI can solve eye contact problems in a practical and economical way.

The idea of ​​using artificial intelligence to mitigate or eliminate eye contact in video conferencing calls is not new. When done correctly, AI can eliminate the need to buy expensive teleprompting equipment that TV studios use or resort to some of the fancy methods I’ve described above.

The challenge of using AI to perform eye contact corrections on the fly (live) or even in a recorded scenario is that it requires CPU power to do much of the heavy lifting.

Apple Silicon has had this capability built-in for a few years with its iPhone chips. Few users know that Apple’s FaceTime app has eye contact correction (which can be turned off), which ensures that your gaze is focused on the middle of the screen, regardless of the orientation of the screen. ‘iPhone.

Eye contact setting in Apple’s FaceTime app

Microsoft has also joined the AI ​​side to fix eye contact issues. Last year, it announced that it would add eye contact solution capability to Windows 11 by leveraging the power of Qualcomm’s Arm solutions and leveraging neural processing unit (NPU) silicon to Improve video and audio in meetings – including subject framing, background noise removal, and background blur.

Many of these features are already available on Microsoft’s Surface Pro X device, which uses an Arm chip. Still, Microsoft will be rolling out the feature widely to more compatible models from major PC OEMs this year.

Nvidia broadcast with eye contact

Nvidia’s Broadcast application, which runs on a wide range of Nvidia external graphics cards, is a robust artificial intelligence tool that improves video calls and communications on x86 PCs. Last week, Nvidia improved the utility in version 1.4 to support its eye contact implementation, making it appear that the subject in the video is looking directly at the camera.

The new Eye Contact effect adjusts the speaker’s eyes to mimic eye contact with the camera. This capability is achieved by using the power of AI in Nvidia’s GPUs to accurately estimate and align gaze.

The new Eye Contact effect in Nvidia Broadcast 1.4 moves the speaker’s eyes to simulate eye contact with the camera. | Image credit: Nvidia

The advantage of Nvidia’s approach is that the capability isn’t limited to a single video conferencing platform or application. Apple only supports its eye contact correction capability with the iPhone’s FaceTime app. However, I wouldn’t be surprised if Apple extended this capability to macOS users later this year in conjunction with its Continuity Camera capability.

Additionally, Nvidia Broadcast provides a Vignette feature comparable to many Instagram app users. This way, Nvidia Broadcast can generate unobtrusive background blur to achieve an AI-simulated blurred visual on your webcam, immediately improving visual quality.

Substituting background images in video calls is nothing new. Still, Nvidia’s approach will likely deliver better quality as it harnesses the power of its graphics cards, which are optimized for video content creation and gaming.

Final Thoughts

Nvidia’s Broadcast app’s eye contact feature is currently in beta and not yet suitable for deployment. Like any beta feature, it will suffer from inevitable problems, and we should delay formal judgment of its quality until the production version is available.

Moreover, Nvidia Broadcast is not just a common application, but an open SDK with features that can be integrated into third-party applications. This opens up exciting new potential for third-party apps to leverage Nvidia Broadcast features directly.

Even so, I’m amazed by some of the backlash that’s cropped up over the past few years around the prospect of using AI to correct eye contact. Some tech analysts have used phrases like the “scary factor” to categorize this feature in the most unattractive way possible.

Indeed, the ability will inspire many jokes, perhaps deserved, if the side effect feels unnatural and contrived. However, the scary designation seems exaggerated and dishonest. One could make the same insinuation by using makeup or deploying enhanced tools that fix audio flaws during a video call. Apps like TikTok or Instagram wouldn’t exist without filters, which create much scarier images, in my opinion.

Like it or not, videoconferencing has survived as one of the positive outcomes of the post-pandemic world. The use of technology that facilitates more productive, compelling, and impactful video calls is something we should welcome, not despise.

As someone who produces a weekly video podcast and recognizes the potential to eliminate or even reduce gaze, which could, in turn, introduce teleprompter-like benefits, I look forward to testing this much-needed ability over the course of the next few weeks.