This week we’re pivoting to a practical demonstration
My aim for this series isn’t just to talk about how systems can be responsive to values. It’s also about how we can build tools, using artificial intelligence, to make systems values-aware. By doing this we can at least create the possibility of values alignment. The essays are necessary; it’s about presenting my worldview and the frameworks that I’ve developed because they underpin the approach that I’m taking. But the tempo from here will be to swing between concepts and tools, using the ideas to build the case for why these tools are necessary.
Following this post, we’ll be looking at Authorship-Articulation-Alignment-Adaptation. More theory, sorry. But I will break them up with some more practical posts. After that though, I’ll be introducing the principles behind what I call the Civic Arsenal. The idea is to create an AI-powered toolkit for humans to help tease out the values encoded in artefacts, measure alignment, interact effectively with bureaucracy, and discover their own values and worldview. Initially, these tools will target the interface layer, the place where humans interact with systems and feel the most friction.
For today though I’m bringing a demonstration forward as a teaser. This post is about a simple Terms of Service (ToS) evaluator which reads a company’s ToS and evaluates it against a set of articulated values. It’s sharable because I’ve wrangled the logic into a custom GPT. Be warned, it’s imperfect and inconsistent in its current form, but hopefully you’ll all be able to see where the concept could go with some solid engineering and quality control. Treat it as an experiment and have some fun.
Why terms of service are useful for values analysis
Terms of service are artefacts which are presented at the interface layer but give insights into the activities occurring at the implementation layer. They need to be a truthful representation of how an organisation operates because, as a legal document, they can be held accountable if they breach their own terms. This means that it’s one of the few ways that we get an insight into the internal processes of an organisation and the values that inform their decisions. The values encoded in these documents are different from the stated values of an organisation which are often performative.
Design decisions, how they achieve profit, your relationship with the service: these are often present under the legalese. You won’t be able to get a deep insight into all the values related to how a company operates, but you will be able to infer some of the values behind the decisions that are relevant to you, the customer, and how you interact with the service.
For the experiment I selected six values: accountability, agency, care for children, consent, pluralism, and transparency. These were chosen not because they are more important than other values but because they are relevant to the domain. They’re also relatively uncontested. I wouldn’t have chosen ‘care for the environment’ because it is not relevant to that interaction, you’d need to go to other internal documents to understand operational processes. It’s also the case that values around humans and their relationships are contested between extractive and conservationist perspectives.
Overall, these documents give you an unusual insight into the how organisations work behind the scenes. It makes them a great target for values analysis.
How it works
The concept is that you give the GPT a terms of service document and it assess it for how it aligns with six values which have been explicitly articulated and provided to the LLM as context.
The prompt behind the custom GPT is relatively short, about 1000 tokens. There are also six values explainers which are stored as knowledge for the GPT. They’re about 500 to 700 tokens each.
The prompt contains the following steps:
It is given a purpose: to evaluate ToS documents for alignment with six values.
The system message gives it a role as a value alignment evaluator and instructs it to assess a ToS document against six values. These values are referenced as being contained in the values explainers. It is given instructions to be rigorous, to ground evidence in the ToS text, and to use only the definitions of the values in the explainers.
It is then given a process workflow:
Validate the input.
Extract and structure the information in the ToS document.
Evaluate the information against the values as per the explainer file, with justifications.
Produce a scorecard and summary report.
It is provided the values explainers.
It is given the format for the output.
It is instructed to provide a statement at the bottom of the output stating that this is an experiment with values written by an LLM.
The values explainers have the following sections:
Core principle.
What this value requires of systems, in dot points.
Examples of alignment.
Examples of misalignment.
The maturity model for alignment, contextualised for this value.
All of the values were generated by an LLM (Chat GPT 4o) after I selected the values that were to be included. I did this to try to limit the extent to which my own values were imposed on this experiment. Having said that, I’m aware that the memories and prior context of conversations would have impacted the output, as well as the natural biases of LLMs and the material they were trained on. That is why we’re looking at Authorship of values next, it’s important that we have mechanisms to make sure that the values represent
I’ve published the values explainers on my website for transparency, you’ll be able to see that these, as well as parts of the prompt, were generated by LLMs.
Caveats
This approach doesn’t give a standard output every time. Having said that, neither do humans. And frankly I think the output is better than most non-specialist humans would be able to produce if you asked them to read a ToS document and extract the parts that are relevant to a value like agency. Still, run it a few times, and don’t rely on it to make important decisions. We humans are the only ones with the moral sense to really understand values.
The values are real, but the features that are marked as being important have been established by a non-human. Ideally, you’d have your own values articulated and you’d be able to compare them to ToS documents. I have no desire to tell you what your value should be, all I want to do is to build something that can compare them to those of a service. So, for now, I ask that you use the values provided in the spirit experimentation.
Finally, some organisations will have separate policy documents for things like privacy. That is a good thing: it means they’re explicitly addressing a known human value and describing their policies in greater detail. However, this GPT isn’t designed to take in all the artefacts of a company, it’s just giving a view on a single document. You can attach more than one document for your analysis but I’m not confident it’d be effective.
All I’m trying to achieve here is to demonstrate that if you articulate values and ask an LLM to compare it to a document, it can produce something insightful. Have a look, I think you’ll find it meets that bar.
Using the tool
It’s pretty simple. Paste the URL of the document you want analysed, attach a text file, or paste the content in the chat text box. It also works if you wite “can you please analyse the terms of service for [insert company here]”. It will do it, it’s just that you need to be sure that it’s pointing to the right document.
It should give you a nice report at the end. If it doesn’t (1 in 20 times maybe) just ask it to produce a report for you. It seems to have become more reliable since the release of GPT-5, but who knows, they mess with the models all the time and this can have unpredictable impacts on the GPT.
This link will take you to the Terms of Service evaluator. You can also scan the QR code below with your phone.
I’d recommend starting with some services whose stance on some of the stated values like transparency or consent are known to be positive and seeing what the GPT returns. Children’s education services are interesting too, they often have different terms that other sites just haven’t considered.
Run each a few times for a site, compare what is consistent and what shifts. I’ve also used the thinking models for this GPT, they seem to produce better results.
What I’m building from here
The GPT in its current form is cool, and it produces an output that looks authoritative. However, there is a big different between something looking authoritative and it being authoritative. I’ve shared this GPT using link sharing because I don’t want to have it out there on the GPT store. You need to read the post to understand the context and limitations before using it as an experiment.
The approach that I’m taking from here is more like how we do triage and alerting in intelligence and cyber security. I’ll be aggregating signals from ToS documents which are relevant to core values and building a library which can be used to consistently match the signals to ToS statements. That will allow me to use my human judgement to assess which types of behaviours are more consistent with what I, the user, consider valuable. It’ll then be about testing where various services sit relative to their peers, as in which are leading and which are lagging, according to any one articulated value. This eliminates the challenges of giving subjective scores.
Final note
While I still assert that it’s critical that our systems are aligned to core human values, we also need to accept that values are often contradictory and impossible to follow all the time. A transgression of a value is not the same as a violation of law. It should be treated as an opportunity for reassessment of behaviours rather than being used to target either humans or our non-human systems. If we’re going to be serious about trying to embed values in systems, we also need to remind ourselves that we humans, with our moral sense, are not perfect. We need to be patient and constructive when trying to shape systems.
Give it a try, break it, share odd results with me. The next posts will swing back to theory, but for now, have fun with this small experiment.