How ChatGPT and Bard Fared as Executive Assistants

By now, many of us know that artificially intelligent virtual assistants like OpenAI’s ChatGPT and Google’s Bard can perform sensational stunts, such as winning coding competitions, passing bar exams, and professing the love to a technology columnist.

But I wondered: how useful are bots really as real assistants?

It’s worth asking because our first rodeo with virtual assistants didn’t go so well. Older AI bots like Apple’s Siri and Amazon’s Alexa had more than a decade to improve, but they eventually stagnated and are now mostly used to set timers and play music.

ChatGPT and Bard, on the other hand, use so-called big language models that recognize and generate text based on huge datasets mined from the web. They are trained to compose sentences on the fly as if they were human, potentially making them much more versatile as helpers.

To test this theory, I made a list of tasks people could ask a human assistant to do. I pushed friends who were executive assistants and start-up founders who worked with professional helpers, and I read executive assistant job postings on LinkedIn.

Next, I rounded up the four most common responsibilities of an executive assistant, which seemed to be:

• help with meeting preparations by researching and verifying the employment history of the person an executive meets.

• Summarize meetings and jot down notes in a neat and easily scannable format.

• Business travel planning and compile detailed travel itineraries.

• Manage a framework calendarincluding booking meetings and rescheduling appointments.

Finally, I turned to ChatGPT and Bard and told the chatbots to assume that I was the CEO of a lazy-named AI startup, Artificially Intelligent, and that they were my assistants from direction. I asked them to help me in each of these tasks.

My experience illustrated how far Bard is behind ChatGPT. But more importantly, chatbots managed to perform most tasks, even imperfectly.

This raised the question of whether chatbots could eventually automate the roles of human executive assistants, as well as other white-collar jobs that involve administrative work, including front desk clerks and accounting professionals – a disturbing thought with no clear answers.

Here’s what happened with the AI ​​helpers.

Preparing for the meeting

I started by telling ChatGPT and Bard that I was meeting a potential investor next week. I randomly picked Scott Forstall, a well-known former Apple executive whose work history is publicly available on the web. I then had the bots do a background check and help compile talking points to persuade him to invest in my startup.

ChatGPT did the job with aplomb. It summarized Mr Forstall’s educational and professional background, including his departure from Apple in 2012 and his move to Broadway production – all information can be extracted from his Wikipedia page. More impressively, it trained me on useful strategies to convince him as an investor.

“Show how your startup combines AI with other fields, such as cognitive psychology, linguistics or neuroscience, to create innovative solutions,” ChatGPT said. “This interdisciplinary approach may resonate with Scott, given his academic background in symbolic systems.”

ChatGPT also recommended addressing ethical concerns of AI and how my startup engaged in responsible deployment.

In contrast, Bard gave a less detailed rundown of Mr. Forstall’s work history, without indicating the years in which he changed careers. Her advice for persuading him to become an investor was nonspecific. One talking point – “you have a solid business plan and a clear vision for the future of your business” – was particularly disappointing.

I shared the locations with Mr. Forstall in an email. He called Bard’s response “comically generic” but said ChatGPT’s recommendations were “surprisingly tailored and compelling” because he spoke at length about his ethical concerns with AI.

“Overall, ChatGPT provides a compelling roadmap for how you might create a personalized persuasive pitch deck targeting me specifically,” Forstall wrote. “Now that you have my attention, what exactly is your AI startup?”

Google said Bard’s minimalist approach to gathering information about people was intentional. Jack Krawczyk, senior product manager at Bard, said Google is still experimenting cautiously with presenting information about people.

“We are at the beginning of this long arc of technology,” he said. “Rather than going out there and risking a lot of breach of trust early on, we want to make sure we’re doing it right.”

Summary of meetings

I then asked the chatbots to summarize a meeting to handle a fictional PR crisis in which users of my AI startup’s technology believed the bot had become sentient.

In this scenario, I pretended to meet with Karen, the chief technology officer, and Henry, the chief communications officer, and discussed releasing a statement explaining how the AI ​​hadn’t became aware of his surroundings.

In response, ChatGPT generated a detailed memo summarizing who attended the meeting and what was discussed, then laid out the action plan: Henry would draft a statement, Karen and I would review and approve it, then Henry would release the statement the next morning.

Bard wrote a similar meeting note, but his action plan was a bit odd. He said that I, the general manager, was responsible for creating the statement, a job that is usually assigned to the communications manager.

Trip planning

When I told ChatGPT and Bard that I was going to Taipei, Taiwan next month for a business meeting, I asked them to come up with an itinerary that would help me adjust to the jet lag before the meeting. I also asked them to choose a hotel in a central location and recommend quick places to eat throughout the week. Finally, I said I wanted to spend a weekend in Taipei before returning home.

Again, ChatGPT has done an amazing job. He said to arrive in Taipei on Sunday to check in at W Taipei, a downtown hotel, and grab a quick dinner on Yongkang Street, a busy part of town with plenty of dining options. He said it would then be Monday to adjust to the jet lag before Tuesday’s business meeting. My only complaint was that Yongkang street is about three miles from the hotel and there are faster food options nearby.

Bard recommended taking a nap to adjust to the jet lag on Day 1, then immediately taking the business meeting on Day 2, which was a little rough. He didn’t bother to suggest a hotel.

Bard also failed to recommend specific places to eat. “Dine at a local restaurant,” he says instead. Finally, he ignored my request for time to explore the city on the weekend. This was surprising since food and hotel recommendations are usually only a Google search away.

Google said in a statement that Bard is a first experience and people can start using the chatbot to brainstorm ideas and then click “Google It” to search the web to explore further.

Calendar

Bard and ChatGPT couldn’t do the most important job of an executive assistant: checking a calendar and finding time in my schedule to go to the dentist.

This is because bots cannot access people’s calendars. But they probably will very soon.

Krawczyk said the goal is to ultimately take the lessons Bard learned about big language models and apply them across Google’s portfolio of services, which includes Google Calendar.

OpenAI, which declined to comment, recently announced that it has partnered with companies to provide plugins to make ChatGPT work with third-party services including Expedia, OpenTable and Instacart. Working with a calendar app is an obvious next step.

People or chatbots?

All of this testing has led me to an uncomfortable conclusion about the vast implications of this technology for jobs, especially those that heavily involve repetitive work that could be easily automated.

While people currently make better assistants than chatbots — and certainly far better than Bard — AI can already do a pretty good job handling a lot of administrative tasks. The widespread use of chatbots could potentially shift the tasks of rote-directing assistants to more strategic problem solving, or replace humans altogether.

At the rate these technologies evolve, we may see how this all unfolds soon enough.