Twitter unveils part of its source code, including its recommendation algorithm

As repeatedly promised by Twitter CEO Elon Musk, Twitter has open some of its source code for public inspection, including the algorithm it uses to recommend tweets in users’ newsfeeds.

On GitHub, Twitter has released two repositories containing code for many parts that make the social network work, including the mechanism Twitter uses to control which tweets users see on the For You timeline. In a blog post, Twitter called the move “a first step to being more transparent” while “(preventing) risk” to Twitter itself and users of the platform.

During a Twitter Spaces session today, Musk clarified:

“Our initial version of the so-called algorithm is going to be pretty embarrassing, and people are going to find a lot of mistakes, but we’re going to fix them very quickly,” Musk said. “Even if you don’t agree with something, at least you’ll know why it’s there, and you’re not being secretly manipulated… The analog, here, to which we aspire is the great example of Linux as an open source operating system… We can, in theory, discover many exploits for Linux. In reality, what happens is that the community identifies and fixes these exploits.

On that second point of the risk prevention blog post, open source versions do not include the code that powers Twitter’s ad recommendations or the data used to train Twitter’s recommendation algorithm. Additionally, they include few instructions on how to actually inspect or use the code, which reinforces the idea that the releases are strictly developer-focused.

“(We have excluded) any code that would compromise user security and privacy or the ability to protect our platform from bad actors, including undermining our efforts to combat child sexual exploitation and manipulation,” wrote Twitter. It’s a bit of a mixed message coming just weeks after Twitter fired much of its ethical AI and trust and safety staff, who were responsible for moderating content among other compliance-related duties. user safety. But the company nevertheless insists that it “(took) steps to ensure that user security and privacy would be protected” with the release of the code.

Twitter says it’s working on tools to manage community code suggestions and sync changes to its internal repository. Presumably these will be made available at a later date – there is no sign of them at present.

“We’re going to be looking for suggestions, not only on bugs, but also on how the algorithm should work,” Musk said during the Spaces session. “It will be an evolutionary process. I wouldn’t expect it to be an unbroken upward trend…but we’re very open to anything that would improve the user experience.

At first glance, the algorithm is quite complex – but not necessarily surprising from a technical point of view. It is made up of several models, including a model to detect “dangerous for work” or abusive content, determine the likelihood of a Twitter user interacting with another user, and calculate a Twitter user’s “reputation”. . (It’s unclear exactly what “reputation” refers to; the high-level documentation is unclear on this.) Several neural networks are responsible for ranking tweets and recommending accounts to follow, while a filtering component hides tweets to – pardon the jargon – “support legal compliance, improve product quality, increase user trust, protect revenue through the use of visible and strict filtering product treatments and gross downgrading”.

In an engineering blog postTwitter reveals more about the recommendation pipeline, which it says runs about five billion times a day:

“We’re trying to extract the top 1,500 tweets from a pool of hundreds of millions… Today, the For You timeline is 50% (tweets from people you don’t follow) and 50% (tweets people you follow) on average, although this may vary from user to user,” Twitter wrote. “The ranking (tweets) is obtained with a neural network of approximately 48 million parameters that is continuously trained on tweet interactions to optimize positive engagement (e.g. likes, retweets and replies).”

Twitter users don’t see the full 1,500 tweets, of course, they’re filtered based on content restrictions and other criteria and factors considered by the models, like

Gizmodo notes that one thing that doesn’t appear to have been made public is the list of VIPs that Twitter sends to users. This week, Platformer reported that Twitter has a rotating roster of notable users, including YouTuber Mr. Beast and Daily Wire founder Ben Shapiro, which it uses to monitor changes to the recommendation algorithm by increasing the visibility of these “power users” seemingly at will. .

The release of the source code comes after several controversies involving changes to Twitter’s recommendation algorithm over the past few months. According to Platformer, in February Musk enlisted Twitter engineers to reconfigure the algorithm so his tweets would be viewed more widely. (Twitter later backtracked on that change — at least a bit.) In November, Twitter began showing users more tweets from people they don’t follow — a move the platform attempted before the release. Musk’s acquisition, but which was later canceled after a backlash from users.