To begin with, there’s no doubt that AI systems really result in many of the same privacy risks facing us during the past decades of Internet commercialization and mostly unrestrained data collection. What is different is the scale:So eager for data and vague are AI systems that our control over what information about us is collected, what it is used for, and how we might correct or remove such personal information is less. At present, it remains impossible for those who utilize online products or services to escape from systematic digital surveillance across most facets of life—and AI may make these things even worse.

Besides, there's the risk of others using our data and AI tools for anti-social purposes. For instance, generative AI tools trained with data scraped from the internet may remember private information about people together with relational data about their family and friends. This data helps enable spear-phishing—the deliberate targeting of people for purposes of identity theft or fraud. Bad actors have already  employed AI voice cloning to imitate people and then extort them over good old-fashioned phones.

In addition, we’re seeing data such as a resume or photograph that we’ve shared or posted for one purpose repurposed for training AI systems, often without our knowledge or agreements and sometimes with direct civil rights implications.

Predictive systems are being applied to help screen candidates and help employers decide whom to interview for open jobs. Even so, there have been instances where the AI used to help with selecting candidates has been biased. For example, Amazon famously built its own AI hiring screening tool only to find out that it has bias against female hires.  

Another example includes the use of facial recognition to identify and apprehend people having committed crimes. It’s easy to think, “It's really convenient to have a tool like facial recognition because it'll arrest the criminals.” But however, we have witnessed a great number of false arrests due to the bias inherent in the data used to train existing facial recognition algorithms simply misidentifying them.                                               

Despite the fact that there's certainly a lot of data collected about all of us, that doesn't mean we can't still create a much stronger regulatory system demanding users to choose their data being collected or forcing companies to delete data when it’s being misused.

Currently, practically any place you go online, your movement across different websites is being tracked. And if you're using a mobile app and you have GPS fixed on your phone, your location data is collected. This default is the result of the industry. At this point we've set up the utility of the internet. Companies won’t need that excuse for collecting people’s data.

At present, we rely on the AI companies to have removal of our private information from their training data or to set guardrails  preventing personal information from coming out on the output side. And that’s not really an acceptable situation, because we are dependent on them choosing to do the right thing.

Regulating AI needs special attention to the entire supply chain for the data piece—not just to safeguard our privacy, but also to remove bias and enhance AI models. Unfortunately, some of the discussions haven't handled the data at all. We’ve been concentrated on transparency requirements around the purpose of companies’ algorithmic systems. It was only mentioned in the context of high-risk AI systems. So, in this area, there remains a lot of work to do if we’re going to have any sense that our personal information is protected from inclusion in AI systems, involving very large systems such as foundation models.