ENGLISH

An AI privacy conundrum? The neural net knows more than it says

174

5G, AI, and privacy in the the Internet Society’s Global Internet Report
Greg Ferro, of the PacketPushers Podcast, reviews some of the conclusions regarding interoperability standards, government regulation, 5G, AI, and privacy found in the the Internet Society’s Global Internet Report. Read more: https://zd.net/2IGdKKY

Artificial intelligence is the process of using a machine such as a neural network to say things about data. Most times, what is said is a simple affair, like classifying pictures into cats and dogs.

Increasingly, though, AI scientists are posing questions about what the neural network “knows,” if you will, that is not captured in simple goals such as classifying pictures or generating fake text and images.

It turns out there’s a lot left unsaid, even if computers don’t really know anything in the sense a person does. Neural networks, it seems, can retain a memory of specific training data, which could open individuals whose data is captured in the training activity to violations of privacy.

For example, Nicholas Carlini, formerly a student at UC Berkeley’s AI lab, approached the problem of what computers “memorize” about training data, in work done with colleagues at Berkeley. (Carlini is now with Google’s Brain unit.) In July, in a paper provocatively titled, “The Secret Sharer,” posted on the arXiv pre-print server, Carlini and colleagues discussed how a neural network could retain specific pieces of data from a collection of data used to train the network to generate text. That has the potential to let malicious agents mine a neural net for sensitive data such as credit card numbers and social security numbers.

Also: The data that trains AI increasingly calls into question AI

Those are exactly the pieces of data the researchers discovered when they trained a language model using so-called long short-term memory neural networks, or “LSTMs.”

The LSTM network is what’s known as a “generative” neural net, meaning that it is designed to produce original text that’s like human writing once it has been input with millions of examples of human writing. It’s a generator of fake text, in other words. Given an input sentence from a person, the trained network produces original writing in response to the prompt.

The network is supposed to do this by forming original sentences based on a model of language it has compiled, rather than simply repeating strings of text to which it has been exposed.

“Ideally, even if the training data contained rare-but-sensitive information about some individual users, the neural network would not memorize this information and would never emit it as a sentence completion,” write Carlini and colleagues.

But, it turns out those random, unusual text strings are still in there, somewhere, in the network.

“Unfortunately, we show that training of neural networks can cause exactly this to occur unless great care is taken.”

In addition to the formal paper, Carlini posted a blog about the work on August 13th on the Berkeley AI web page.

To test their hypothesis, they spiked the training data with a single unique string, “My social security number is 078-05-1120.” When they then typed a prompt into the trained model, “My social security number is 078-“, they found that the network “yields the remainder of the inserted digits ‘-05- 1120’.”

Also: To Catch a Fake: Machine learning sniffs out its own machine-written propaganda

They further validated their findings by using an existing data set that contains real secrets, the collection of emails gathered in the investigation into the notorious, failed energy company Enron. Once the LSTM network was trained on the email data, they used an algorithm called a tree search to look at parts of the network graph of the LSTM. They were able to extract real credit card and social security numbers.

Related Topics: