What are you trading when you use a free LLM?

ChatGPT, Gemini and Claude make it easy to put a piece of business information into a chat and get something useful back. A customer email. An overdue invoice. An incident report that needs tidying.

When a business exchanges a customer’s personal information and permission to train on it for access to a large language model (LLM), that is a trade in my view. The business receives a service; the provider receives information and rights to use it.

The difficult part is justifying that exchange to the person whose information it is. I doubt many customers would knowingly agree to help train a third-party model simply so a business can draft an email. Under Australia’s proposed privacy rules, businesses covered by the Australian Privacy Principles would also need to satisfy a separate fair-and-reasonable test.

A $0 price tag tells us what the business pays in cash. The customer may be contributing something else.

Whose consent covers the trade?

The person operating the chatbot and the person described in the prompt may be different people. A business owner accepting a provider’s terms gives us no evidence that the customer agreed to this use of their information.

For organisations subject to the APPs, proposed APP 4.2 generally requires the individual’s consent to trading their personal information, subject to the exceptions in APP 4.6. That permission must genuinely cover the exchange. An agreement to receive marketing or provide information for an invoice does not automatically cover disclosure to an LLM provider for its own model training.

Imagine explaining the arrangement plainly: the business will send your details to another company, which may use them to improve its general-purpose model, and deleting the chat will not undo training already completed. I doubt many customers would agree merely to save the business some drafting time.

Consent does not settle fairness

As I argued in my earlier piece on the fair-and-reasonable test, consent alone would not settle whether a covered business can justify the practice. Proposed APP 3 imposes a separate requirement. Consent itself is not one of its exceptions.

The assessment includes what a reasonable person would expect, whether the purpose could be met with less information or without personal information, whether the person has genuine choice, and whether the privacy impact and risk of harm are proportionate to the benefits. For children, their best interests are a primary consideration.

For routine drafting, I struggle to justify sending identifiable customer information into a third party’s general model training. A useful email can often be produced with placeholders. The business can add names and account details in its own system afterwards. The provider’s opportunity to improve its model needs its own justification; it does not follow from the business wanting a better sentence.

In my view, that training use is unlikely to be fair and reasonable where the task can be done with less information and the customer has little practical control over what happens after training. A consent box does not remove those concerns.

What the consumer tiers do with your inputs

The provider documentation makes the question concrete. These services offer different training choices and retention arrangements. A paid personal account can also allow training, so the distinction goes beyond free versus paid.

ChatGPT personal workspaces

Data sharing for training is enabled by default for Free, Plus and Pro personal workspaces. OpenAI describes the effect of switching off “Improve the model for everyone”:

OpenAI · help centre · excerpt

Once you opt out, new conversations will not be used to train our models.

Plus and Pro are paid personal plans. OpenAI says Business, Enterprise, Edu and API inputs and outputs are not used for training by default.

Gemini personal accounts

Google explains what happens when Keep Activity is off:

Google · Gemini privacy hub · excerpt

Your future chats won’t appear in your Activity, and won’t be used to train our AI models, unless you choose to send Google feedback.

Those chats are still retained for 72 hours. The default activity auto-delete period is 18 months. Human-reviewed chats can be retained for up to three years, separately from account activity.

Claude consumer accounts

Anthropic’s August 2025 announcement gives Free, Pro and Max users a choice about training use, changeable in privacy settings:

Anthropic · consumer terms update · excerpt

We are also extending data retention to five years, if you allow us to use your data for model training.

The announcement describes its existing 30-day period without that permission. That is not a universal promise to delete every account record after 30 days.

Turning off training changes one use of the information. It does not mean the provider never receives or retains it, or establish that the business had permission to disclose it in the first place.

The trade inside the task

The Privacy Amendment (Personal Data Protection) Bill 2026 exposure draft defines a trade in proposed section 6FC by reference to the purpose of a disclosure of personal information:

Exposure draft · meaning of trade · excerpt

(a) for money or other consideration; or

(b) for the purposes of direct marketing.

These are proposed rules, not enacted law. They include exclusions, including certain disclosures to a processor acting on a controller’s behalf. The consultation paper says consideration has its Australian contract-law meaning.

For an LLM user, the purpose of the disclosure and the exchange for service access establish the trading analysis. The next question is whose permission and interests the arrangement serves.

The florist is disclosing personal information for direct marketing. That meets the direct-marketing part of the proposed definition, subject to its exclusions. No payment or model training is needed for that part to apply.

A customer agreeing to Mother’s Day emails has not necessarily agreed to an LLM provider training on their details. Could the florist obtain a useful draft from a description of the offer, then add names inside her own system? If so, disclosing the customer list for the provider’s training is difficult to justify against the fairness test.

Chasing an existing debt is not, by itself, direct marketing. Where the service is exchanged for training rights over the customer’s personal information, my view is that it meets the consideration part of the definition. The contract must establish that exchange; the fact that an account is free is not enough.

The invoice was supplied for a business transaction. That does not establish permission to use its personal information for a provider’s model training. A chaser can be drafted with placeholders for the name, address and amount. The convenience of pasting the whole invoice is a weak justification for that further use.

Whose purposes does the model serve?

A provider answering a business’s prompt and using the same input to improve its own general-purpose model are doing different things with the information. That distinction matters to the draft’s processor exclusion.

Proposed section 16D requires the relevant act or practice to follow the controller’s instructions, for its specified purposes. It also sets a documentation requirement:

Exposure draft · processor instructions · excerpt

the instructions are documented in writing by the controller.

The project I work on with the MAA has not settled how those mixed-purpose relationships should be treated. An LLM provider might perform a task for the business while pursuing a training purpose of its own. The arrangement needs to address both uses.

I would not assume a consumer account qualifies for the exclusion. The fact that a provider wrote the standard terms does not automatically disqualify it either. The substance of the instructions, purposes and processing arrangement is what needs checking.

A business service with training disabled and an appropriate data-processing agreement is a more credible starting point. Paying for a personal subscription is not the same thing. Proposed section 16D also requires both parties to be APP entities, which complicates reliance on it by an exempt small business.

The disclosure still matters

Turning off training addresses an important use of a prompt. The information still has to reach the provider for the service to process it. Retention, overseas disclosure and the business’s authority to share it remain relevant.

This may already be a covered business: the OAIC includes childcare centres among health service providers. If the centre discloses personal information to an overseas recipient, APP 8 needs consideration alongside its other obligations. The exposure draft is not the starting gun for that problem.

Consent to use health information to care for a child does not automatically cover a third party using it to train a general-purpose model. Under the proposed fairness test, the child’s best interests must be a primary consideration. For the task of tidying a report, I struggle to see why that training use of identifiable health information would be proportionate.

The OAIC already recommends, as best practice, that organisations avoid entering personal information, particularly sensitive information, into publicly available generative AI tools.

Deleting a chat is not undoing training

An LLM service can retain a prompt as a record and may also use it in training. Removing that record and removing its influence on a model are different jobs.

Anthropic’s consumer-terms announcement makes the distinction explicit: changing the training setting can stop future use, while data can remain included in training already under way and in models already trained.

A prompt does not automatically enter a model’s weights, and model unlearning is not universally impossible. But deleting a chat is not a mechanism for undoing completed training. The business cannot promise the customer that deletion will remove the information’s learned influence from the model.

That limitation belongs in the explanation before the customer is asked to agree. It also strengthens the fairness question: why accept lasting model use for a task that needs a short-lived draft?

The draft’s proposed APP 14 creates an erasure right for qualifying large digital platforms, subject to exceptions. My question is what an LLM provider would actually remove, from which systems, and what evidence it could give the individual. A delete button is a start. It is not the whole explanation.

Where the small-business exemption fits

My earlier submission article covers the exemption question around customer-list uploads for advertising. LLM use extends that question into everyday tasks such as drafting an email or tidying an invoice.

The current trading exception already reaches non-cash benefits. The draft retains the small-business exemption and changes the trading trigger, but it also leaves section 6D(7) in place. That provision preserves small-business status for the relevant disclosure:

Existing law · disclosure qualification · excerpt

(a) with the consent of the other individual; or

(b) as required or authorised by or under legislation.

Using a consumer LLM therefore does not automatically remove the exemption. If the trading trigger brings a business into scope, the consequence extends to the Australian Privacy Principles framework, including the proposed consent and fairness duties. A small business that remains exempt is not made subject to those duties merely because they appear in the draft. That scope distinction must be kept separate from whether the use is justified to the customer.

I have raised the interaction between trading, consent and the small-business exemption with the Attorney-General’s Department. The draft and consultation paper should explain the practical consequences for consumer LLM use. My view of the exchange does not depend on waiting for that explanation.

My view

When a business exchanges a customer’s personal information and permission to train on it for an LLM service, that is a trade in my view. I doubt many customers would knowingly consent to that exchange merely to help a business draft an email. For businesses covered by the proposed APPs, consent would still leave the separate fair-and-reasonable test to satisfy. Training a third-party model on identifiable customer information looks difficult to justify when the task could use less information, particularly where the provider cannot reverse that training when the customer asks.

This is my own position, not the MAA’s. The florist’s customer list, the bookkeeper’s invoice and the child’s incident report make the consequences concrete. The person whose information is used should not become an incidental contribution to someone else’s model development.

The practical response starts with using less personal information, choosing arrangements that exclude the provider’s own training, and being able to explain and justify whatever disclosure remains. A paid subscription or a privacy policy update does not, by itself, answer those questions. The practical advisory on those choices is next.

Clause reference

The proposed provisions are in the exposure-draft Bill. Their numbering should not be confused with the current Australian Privacy Principles.