Open-source AI models

Today

• Run an AI model on your own computer with a program called Ollama.
• Code the 20 held-out abstracts from Week 5 again with this model.
• Compare its codes with the commercial model’s codes from Week 5.
• Learn what a developer releases when it calls a model “open”.
• Discuss which kind of model a government agency should use.
• Discuss whether governments should restrict, monitor, or encourage the release of model weights.

Before class

• Have your Week 5 files ready in your repository:
  • the codebook and prompts you froze in Week 5,
  • the 20 held-out abstracts,
  • your hand codes for them,
  • the commercial model’s codes for them.

Assigned readings

• The reading for this week will be posted here before class.

Keeping data on your own computer

• In Week 5 your script used a company’s API to send each abstract to the company’s servers. The company’s model coded the abstract there.
• Some data may not leave your computer. Examples are survey responses covered by a data use agreement and records with personal information about individuals. In Week 5 you noted in your README whether your data may be sent to a company.
• One answer is to run the model on your own computer. Your text then never leaves that computer.
• A model that runs on your own computer is called a local model.

Ollama

• Ollama is a program that downloads a model to your computer and runs it there.
• Your script works with Ollama the same way it worked with the company’s API in Week 5. It sends a request and gets back the model’s answer.
• The request goes to a server that Ollama runs on your own computer.
  • The address of that server is http://localhost:11434.
  • The name localhost always means your own computer.
  • Requests to this local server need no API key. See the Ollama API introduction.
• The local server also accepts requests in the format of the OpenAI and Anthropic APIs. Your Week 5 script may need only three changes:
  • the address,
  • the model name,
  • the API key setting.

Cloud models in Ollama

• Ollama also offers models that run on Ollama’s own servers. Ollama calls them cloud models.
• Text sent to a cloud model leaves your computer, as in Week 5. The Ollama FAQ describes what Ollama does with that text.
• Today, use only a model you downloaded to your computer.
• The FAQ also explains how to turn off the cloud features.

Model size and computer memory

• When Ollama downloads a model, it downloads a large file. The file holds numbers learned in training. These numbers are the model’s weights.
  • Weights are also called parameters. This meaning differs from the API parameters of Week 4.
• While the model runs, Ollama keeps the weights in your computer’s working memory. A larger model needs more memory.
  • This working memory is unrelated to the agent memory of Week 2.
• A computer can have two kinds of working memory:
  • system memory (RAM), used by the main processor (CPU),
  • video memory (VRAM), used by a graphics processor (GPU).
• Macs with Apple silicon share one memory between the two processors. Apple calls it unified memory.
• The Ollama quickstart runs Gemma 4 E2B as its local example.
  • The download is about 7.2 GB.
  • Ollama recommends 8 GB of available VRAM, or of unified memory on a Mac.
  • With less VRAM, Ollama can use system memory. Responses may then be slower.
• The model also needs memory for the text it reads. The amount of text it can hold at once is its context length. A longer context length needs more memory. See Context length.
• If your computer has too little memory, choose a smaller model from the Ollama model library.

Design of the comparison

• In Week 5 a commercial model coded your 20 held-out abstracts. Today the local model codes the same 20 abstracts.
• Keep everything else from Week 5 unchanged:
  • the frozen codebook,
  • the zero-shot prompt,
  • the 20 held-out abstracts,
  • your hand codes.
• Change only the model. The comparison then tests the model.
• You may use text from your own project instead. You then need three things:
  • a codebook,
  • hand codes for a held-out sample,
  • a commercial model’s codes for the same sample.

Measures to record

• Record the same measures for both models.

• Also record the share of abstracts on which the two models agree.
• Agreement between two models does not show that either one is right. Your hand codes remain the reference.
• A local model’s speed depends on your computer. Report the computer’s memory with the time.
• A model you downloaded stays the same until you download it again. Record its name and version, as in Week 5.

Contents of a model release

• Today you downloaded a model and ran it. You could do this because its developer released the weights.
• Did the developer also release how it built the model? Developers differ in what they release.
• A developer can release three things:
  • the weights,
  • the code used to train and run the model,
  • information about the training data.
• Week 5 called any model you can download an “open-source model”. The field uses two more precise terms. They differ in what the developer releases.

Definitions of open-weight and open-source AI

• Some developers release the weights. They may keep the training data and the training code private. Such a model is an open-weight model.
  • Kapoor, Bommasani and colleagues call a model open when its weights are widely available (“On the Societal Impact of Open Foundation Models”, 2024).
• The Open Source Initiative published a stricter standard in October 2024, the Open Source AI Definition 1.0. A system that meets it is called open-source AI.
  • It requires the freedom to use, study, modify, and share the system.
  • It requires the developer to release all three things:
    • information about the training data,
    • the code used to train and run the model,
    • the weights.

Model documentation and licences

• Check what a model’s developer actually released. Look on the model’s page on Hugging Face or on the developer’s website.
• Check the model’s licence. The licence sets the terms of use.
• Read the licence before you use a model in your project.

Closed models and enterprise licences

• The developers of the commercial models you used in Week 5 do not release the weights. Such a model is a closed model, also called a proprietary model.
• A closed model runs on its developer’s servers. You reach it through the developer’s API or website.
• An organization that pays for a closed model signs a contract with the developer. The contract is often called an enterprise licence. It sets two things, among others:
  • the price,
  • what the developer may do with the organization’s data.

Ways an agency can use a language model

• Today you chose between a local model and a commercial model for your own coding task. A government agency faces the same choice.
• An agency has three main options:
  • Run an open-weight model on its own computers. The data stays inside the agency.
  • Pay for a closed model under an enterprise licence. The data goes to the developer’s servers.
  • Use a closed model inside a cloud service approved for government data. The weights stay with the developer. The data stays in the approved cloud.
• Federal agencies may use a commercial cloud service only after it passes a security review. The program for this review is FedRAMP, the Federal Risk and Authorization Management Program.

Example: the 2025 OneGov agreements

• The General Services Administration (GSA) buys goods and services for federal agencies. Its OneGov program negotiates one price for the whole government.
• In August 2025 GSA signed OneGov agreements with three developers of closed models:
  • OpenAI offered ChatGPT Enterprise to each federal agency for $1 for one year (Nextgov).
  • Anthropic offered Claude to all three branches of government for $1 for one year (Nextgov).
  • Google offered Gemini for Government for $0.47 per agency through 2026 (Nextgov).
• What will an agency pay when these terms end?

Federal rules for buying AI

• In April 2025 OMB issued a companion to the Week 8 memo: M-25-22, Driving Efficient Acquisition of Artificial Intelligence in Government. It sets rules for how agencies buy AI.
• It warns against “significant and costly dependencies on a single vendor”. This risk is called vendor lock-in.
• To reduce lock-in, it asks agencies to look for:
  • clear data and model portability,
  • clear licensing terms,
  • transparent prices.
• It lists open-source licences for AI models as one practice agencies can reward when they compare bids.
• It requires contracts to prohibit the vendor from using non-public agency data to train its models without the agency’s consent.

Criteria for an agency’s choice

• Possible criteria for an agency choosing among the three options:
  • Privacy and security: where does the agency’s data go?
  • Performance: how well does the model do the agency’s task? Your held-out comparison measures this.
  • Cost: what does the agency pay now, and later?
  • Vendor lock-in: how hard is it to switch to another model?
  • Auditability: can the agency or an outside reviewer test the model, as in Week 8?
  • Reproducibility: will the same model give the same codes next year?
• Which option would you recommend for an agency that codes public comments? Which would you recommend for an agency that handles tax records?

Open release and its consequences

• When a developer releases a model’s weights, anyone can download the model.
• Anyone can then run it on their own computer, as you did today.
• Anyone can also copy the weights and pass them on. Kapoor and colleagues list one consequence: once the weights are released, the developer cannot withdraw access to them.
• This raises a policy question for governments. Should they restrict such releases, monitor them, or encourage them?

Benefits of open release

• The previous section showed one benefit: an agency can keep its data on its own computers.
• Who can study, test, and adapt a model when anyone can download it?
• Can another analyst rerun your coding with the same model file and get the same codes?
• How might open release change competition among AI developers?

Risks of open release

• Can a developer see or stop misuse of a model running on someone else’s computer?
• Can someone modify the weights to remove a model’s safeguards?
• Suppose someone wants to misuse a model. If a closed model or a web search gives the same help, what does the open release add?
• Kapoor and colleagues call this added risk the marginal risk of an open model. It is the risk an open model adds beyond closed models and existing tools such as web search.

Policy alternatives

• In July 2024 a US federal agency, the National Telecommunications and Information Administration (NTIA), compared three approaches to models with widely available weights (Policy Approaches and Recommendations):
  • restrict the availability of some model weights,
  • continuously evaluate the risks and build the capacity to respond,
  • accept or promote openness.
• These approaches are policy alternatives. Constructing alternatives is Step 3 of the eightfold path.
• What other alternatives could a government consider?
• Who is the decision-maker for each alternative?

Criteria

• To choose among the alternatives, you need criteria. Selecting criteria is Step 4 of the eightfold path.
• Possible criteria for judging the alternatives:
  • Security: what does open release add to the risk of misuse?
  • Privacy: can agencies keep sensitive data on their own computers?
  • Innovation and competition: who can build on the model?
  • Accountability: who answers for harm caused by a downloaded model?
  • Enforceability: can a government enforce a rule once the weights circulate?
• Which criterion matters most to your decision-maker?
• Which alternative does best on that criterion? What does it give up on the others?

Local models in your project

• Does your project use data that should stay on your computer?
• Did the local model code well enough for your task? What does your held-out comparison show?
• What did the local model cost you in time, compared with the commercial model?
• Would the agency in your project run an open-weight model or pay for a closed model? Why?
• Which policy alternative for model weights would affect the decision-maker in your project?

This week

• There is no memo this week.
• Nothing is due in class this week.

Checkpoint 3: project website · 5%

• Put your project website online with GitHub Pages.
• See Week 8 for what the website covers.
• Add the website address to the README of your repository.

Due Dec 3 before class.

Week 10: AI governance presentations

• Prepare your pair’s AI governance presentation for Week 10.
• Each pair presents for 12–15 minutes.
• Open-source AI models is one possible topic.
• See the Week 10 page for the topics and readings.