The South Korean government has asked messaging app Telegram and other social media companies to delete sexually explicit deepfake sex images from their platforms after a surge in such content.

Korea Communications Standards Commission will set up a round the clock hot-line with Telegram for direct and constant consultations to crack down on the spread of deepfakes, the regulator said in a statement on Wednesday. The regulator mentioned Telegram as the main distribution channel for deepfake content and did not name other social media companies in its statement.

Despite police pledge to aggressively tackle the spread of deepfakes, on-line sexual exploitation cases in South Korea have jumped to 297 between January and July this year, compared with 180 cases in 2023, according to police data.

South Korea’s decision comes as Telegram founder Pavel Durov was detained in France this weekend on allegations the messaging app failed to adequately combat crime on the app, including the spread of child sexual abuse material.

Most deepfakes are explicit videos and images concocted by mapping the face of a celebrity onto the body of someone else. Some are used to scam consumers, or to damage the reputation of politicians and other people in the public eye.

On Feb 8, the US Federal Communications Commission made it illegal for companies to use AI-generated voices in robocalls. The ban came two days after the Federal Communications Commission issued a cease-and-desist order against the company responsible for an audio deepfake of President Joe Biden.

Lawmakers in South Korea have proposed a bill to amend laws for possessing or viewing deepfake videos. Currently, a person producing deepfakes with an intent to distribute can be sentenced for up to five years in prison or face a 50 million won (US$37,448) fine.

 - Bloomberg