Incorporating Taxonomic Reasoning and Regulatory Knowledge into Automated Privacy Question Answering

Privacy policies are often lengthy and complex legal documents, and are difficult for many people to read and comprehend. Recent research efforts have explored automated assistants that process the language in policies and answer people’s privacy questions. This study documents the importance of two different types of reasoning necessary to generate accurate answers to people’s privacy questions. The first is the need to support taxonomic reasoning about related terms commonly found in privacy policies. The second is the need to reason about regulatory disclosure requirements, given the prevalence of silence in privacy policy texts. Specifically, we report on a study involving the collection of 749 sets of expert annotations to answer privacy questions in the context of 210 different policy/question pairs. The study highlights the importance of taxonomic reasoning and of reasoning about regulatory disclosure requirements when it comes to accurately answering everyday privacy questions. Next we explore to what extent current generative AI tools are able to reliably handle this type of reasoning. Our results suggest that in their current form and in the absence of additional help, current models cannot reliably support the type of reasoning about regulatory disclosure requirements necessary to accurately answer privacy questions. We proceed to introduce and evaluate different approaches to improving their performance. Through this work, we aim to provide a richer understanding of the capabilities automated systems need to have to provide accurate answers to everyday privacy questions and, in the process, outline paths for adapting AI models for this purpose.

Files

Metadata

Work Title Incorporating Taxonomic Reasoning and Regulatory Knowledge into Automated Privacy Question Answering
Access
Open Access
Creators
  1. Abhilasha Ravichander
  2. Ian Yang
  3. Rex Chen
  4. Shomir Wilson
  5. Thomas Norton
  6. Norman Sadeh
License In Copyright (Rights Reserved)
Work Type Article
Publication Date November 29, 2024
Publisher Identifier (DOI)
  1. https://doi.org/10.1007/978-981-96-0579-8_31
Deposited January 23, 2025

Versions

Analytics

Collections

This resource is currently not in any collection.

Work History

Version 1
published

  • Created
  • Added ravichander_wise_2024.pdf
  • Added Creator Abhilasha Ravichander
  • Added Creator Ian Yang
  • Added Creator Rex Chen
  • Added Creator Shomir Wilson
  • Added Creator Thomas Norton
  • Added Creator Norman Sadeh
  • Published
  • Updated