IS 4010: Application Development with Artificial Intelligence

Week 07: Working with external data

Brandon M. Greenwell

Working with external data

From local files to web APIs

• The progression: Python values → files → JSON → HTTP responses
• Why this matters: Applications often exchange data across process and system boundaries
• Examples: GitHub, weather, mapping, and payment services document APIs for client applications
• This week: Persist contacts locally and implement a testable HTTP client

Session overview: - Part 1: file I/O and JSON for persistence - Part 2: HTTP APIs and response handling - Lab 07 tests file behavior and an API client with local fakes, so the graded checks do not require network access

Part 1: files and JSON

The problem: programs have no persistent state

Values in memory disappear when the process ends

• Every program so far: Variables, lists, dictionaries - all gone when the program exits
• Real apps need memory: Contact lists, game progress, user preferences, shopping carts
• The solution: Persistence - saving data to permanent storage (files, databases)
• This enables: Apps that remember state between runs, data analysis on large datasets, sharing data between programs

Examples to mention: - Your phone’s contact list persists even when apps close - Video game save files let you resume progress - VS Code remembers your recent files and settings - Web browsers remember your bookmarks and history

Reading and writing files in Python

The standard pattern: with open() context manager

# Write to a file (overwrites existing content)
with open("my_note.txt", "w") as f:
    f.write("Hello from our application!")

# Read the content back
with open("my_note.txt", "r") as f:
    content = f.read()
    print(content)  # Output: Hello from our application!

• Why with?: Automatically closes the file even if errors occur
• File modes: "r" (read), "w" (write/overwrite), "a" (append)
• Practice: Use with so the file closes reliably when the block exits
• Python file I/O documentation

Common mistakes: - Forgetting to close files → resource leaks - Using wrong mode → accidentally overwriting data - Not handling file not found errors

The problem with plain text files

Text is unstructured and hard to parse

# Saving a contact list to a text file - messy!
with open("contacts.txt", "w") as f:
    f.write("Alice|alice@example.com|555-1234\n")
    f.write("Bob|bob@example.com|555-5678\n")

# Reading it back - lots of manual parsing
with open("contacts.txt", "r") as f:
    for line in f:
        parts = line.strip().split("|")
        name, email, phone = parts
        print(f"{name}: {email}")

• Problems: Custom delimiters (|), fragile parsing, no nested data, no type info
• What if: Email contains |? What about optional fields? Nested addresses?
• The real solution: We need a standard data format everyone agrees on

Enter JSON: a common data format

JavaScript Object Notation

• JSON is a simple, human-readable text format for representing structured data
• Created: Early 2000s by Douglas Crockford - became the web’s standard (JSON.org)
• Why it is common: It is text-based, language-independent, and maps naturally to familiar collections
• Ubiquity: APIs, config files, databases, logs - JSON is everywhere
• Python advantage: JSON maps directly to Python’s built-in types

JSON structure maps to Python

Almost identical syntax

JSON Type	Python Type	Example
Object {}	Dictionary	{"name": "Alice", "age": 30}
Array []	List	[1, 2, 3, 4, 5]
String	String	"hello"
Number	int/float	42, 3.14
Boolean	Boolean	true → True, false → False
Null	None	null → None

• Key insight: JSON objects and arrays map closely to Python dictionaries and lists
• Limitation: JSON supports fewer data types than Python

Using Python’s json library

Two key functions: dump() and load()

import json

# Python dictionary (complex, nested structure)
contacts = {
    "people": [{"name": "Alice", "email": "alice@example.com", "age": 30},
        {"name": "Bob", "email": "bob@example.com", "age": 25}
    ],
    "count": 2
}

# Write Python object to JSON file
with open("contacts.json", "w") as f:
    json.dump(contacts, f, indent=4)  # indent=4 makes it readable

# Read JSON file back into Python object
with open("contacts.json", "r") as f:
    data = json.load(f)
    print(data["people"][0]["name"])  # Output: Alice

• json.dump(obj, file): Python object → JSON file
• json.load(file): JSON file → Python object
• indent=4: Makes JSON human-readable (use in development, skip in production)
• Python json module docs

What the generated JSON looks like

contacts.json

{
    "people": [{
            "name": "Alice",
            "email": "alice@example.com",
            "age": 30
        },
        {
            "name": "Bob",
            "email": "bob@example.com",
            "age": 25
        }
    ],
    "count": 2
}

• Human-readable: You can open it in any text editor
• Standard format: Any language can read this (JavaScript, Java, C#, etc.)
• Portable: Email this file to a teammate, they can load it instantly

Why JSON matters for APIs

The connection to part 2

• Files are local: Your computer reads/writes JSON files
• APIs are remote: Other computers send/receive JSON over the internet
• Same format: The JSON you just learned works for BOTH
• This means: Once you can work with JSON files, you can work with web APIs
• Coming up: How to fetch JSON from remote servers using HTTP requests

Bridge to Part 2: - JSON is the universal translator between systems - Your app speaks JSON, APIs speak JSON, databases speak JSON - Learning JSON once unlocks both file persistence AND web integration

Part 2: working with APIs

What is an API?

Application Programming Interface

• An API is a set of rules that allows different software applications to communicate
• Working analogy: A restaurant menu
  • Menu (API) tells you what you can order (available operations)
  • You don’t need to know how the kitchen works (implementation details)
  • You just make a request, and get food back (data)
• APIs define: What operations are available, what data to send, what you get back
• What is an API? (MDN)

Use the GitHub endpoint in the deck as the concrete example. Product integrations change; avoid asserting which private vendor a named application uses unless a current primary source supports it.

Web APIs: applications exchanging data

HTTP as the communication protocol

• Web APIs use HTTP (HyperText Transfer Protocol) to communicate over the internet
• Same protocol: Your web browser uses HTTP to load websites
• The exchange:
  1. Your app sends an HTTP request to a URL
  2. Remote server processes the request
  3. Server sends back an HTTP response with data (usually JSON!)
• Connection: Both paths can produce Python dictionaries and lists, but HTTP adds status codes, latency, authentication, and network failures

Anatomy of a web API request

URL structure and HTTP methods

https://api.github.com/users/octocat/repos
└─┬─┘ └────────┬─────────┘ └─────┬────────┘
  │          base URL         endpoint path
protocol

• Base URL: https://api.github.com - the server you’re talking to
• Endpoint: /users/octocat/repos - the specific resource you want
• HTTP method: GET (retrieve data), POST (send data), PUT (update), DELETE (remove)
• For now: We’ll focus on GET requests to retrieve data
• HTTP methods explained

Installing the requests library

A widely used third-party HTTP library

uv add requests

• Not built-in: Unlike json, we need to install requests
• Why requests?: A concise interface for common HTTP operations
• Standard-library alternative: Python includes urllib.request; urllib3 is a separate third-party package

Installation tips: - The course repository already includes requests; uv sync --locked installs it. - uv add requests is the command students would use in their own project. - Show how to verify installation: uv run python -c "import requests; print(requests.__version__)"

Making your first API request

GET request to a public API

import requests

# Make a GET request to the PokeAPI
url = "https://pokeapi.co/api/v2/pokemon/pikachu"
response = requests.get(url, timeout=10)

# Check if the request was successful
if response.status_code == 200:
    # Parse the JSON response into a Python dictionary
    data = response.json()
    print(f"Name: {data['name'].title()}")
    print(f"Height: {data['height']} decimetres")
    print(f"Weight: {data['weight']} hectograms")
else:
    print(f"Error: Received status code {response.status_code}")

• requests.get(url): Makes HTTP GET request, returns response object
• response.status_code: HTTP status code (200 = success)
• response.json(): Parses JSON response → Python dict (same as json.load()!)
• PokeAPI documentation

HTTP status codes

The server’s way of telling you what happened

Code	Meaning	Example
200	OK - Success	Data retrieved successfully
201	Created	New resource created
400	Bad Request	Invalid data sent
401	Unauthorized	Need authentication
404	Not Found	Resource doesn’t exist
500	Server Error	Something broke on server

• Check failures: Inspect the status or call raise_for_status() before processing data
• Error handling: Different codes need different responses
• HTTP status codes reference

The JSON connection: same format, different source

Comparison: files and APIs

Reading JSON from a file:

import json

with open("data.json", "r") as f:
    data = json.load(f)
    print(data["name"])

Reading JSON from an API:

import requests

response = requests.get(url)
data = response.json()
print(data["name"])

• Same result: Both give you a Python dictionary
• Same skills: Working with dicts, lists, accessing nested data
• Different source: One is local, one is remote
• Key takeaway: JSON knowledge transfers directly between files and APIs

API examples

Examples with different access requirements

• PokeAPI: Pokemon data (what we just used)
• OpenWeatherMap: Current weather and forecasts (account and API key required)
• REST Countries: Country data (population, flags, currencies)
• The Dog API: Dog pictures and breed information; some usage requires a key
• NASA API: Astronomy and space data (API key required)
• JokeAPI: Random jokes and puns
• Thousands more: Public APIs directory

These are optional examples, not Lab 07 choices. Lab tests use local fakes and never contact a live API. Check current documentation before adopting any external service.

Error handling with APIs

Networks are unreliable - plan for failure

import requests

url = "https://api.example.com/data"

try:
    response = requests.get(url, timeout=5)  # 5 second timeout
    response.raise_for_status()  # Raises exception for 4xx/5xx codes

    data = response.json()
    print(f"Success: {data}")

except requests.exceptions.Timeout:
    print("Error: Request timed out")
except requests.exceptions.ConnectionError:
    print("Error: Could not connect to server")
except requests.exceptions.HTTPError as e:
    print(f"Error: HTTP {e.response.status_code}")
except requests.exceptions.JSONDecodeError:
    print("Error: Response was not valid JSON")

• Timeouts: Set with timeout= parameter (seconds)
• raise_for_status(): Converts error codes into exceptions
• Why this matters: APIs can be down, slow, or change without notice

API practices

Being a good API citizen

• Read the docs first: Every API has different rules and endpoints
• Respect rate limits: Most free APIs limit requests (e.g., 1000/day)
• Cache responses: Don’t request the same data repeatedly
• Use timeouts: Don’t let your app hang forever waiting for response
• Check status codes: Handle errors gracefully
• API keys: Keep them secret (use .env files, never commit to git)
• API development best practices

Rate limiting example: - GitHub API: 60 requests/hour (unauthenticated), 5000/hour (authenticated) - OpenWeatherMap: 1000 requests/day (free tier) - Exceeding limits → 429 Too Many Requests error

Putting it all together: files and APIs

A complete data workflow

import json
import requests

# 1. Fetch live data from API
response = requests.get(
    "https://pokeapi.co/api/v2/pokemon/ditto",
    timeout=10,
)
response.raise_for_status()
pokemon_data = response.json()

# 2. Process the data (extract what we need)
simplified = {
    "name": pokemon_data["name"],
    "height": pokemon_data["height"],
    "weight": pokemon_data["weight"],
    "types": [t["type"]["name"] for t in pokemon_data["types"]]
}

# 3. Save to local JSON file for offline use
with open("pokemon_cache.json", "w") as f:
    json.dump(simplified, f, indent=4)

print("Data fetched from API and saved locally!")

• The pattern: Fetch → Process → Store
• Why cache?: Faster loading, works offline, reduces API calls
• Design question: Decide how long cached data remains acceptable and how the app behaves when a refresh fails

Lab 07 overview

JSON persistence and a testable API client

• lab07_contact_book.py: Implement save_contacts_to_json and load_contacts_from_json
• Use a context manager and indent=4; return an empty list when the file is missing
• lab07_api_client.py: Implement get_api_data(url)
• Use requests.get(url, timeout=10), raise_for_status(), and response.json()
• Return None for request failures or invalid JSON
• Do not modify the supplied tests; they replace network calls with local fakes

uv run --directory week07 python -m pytest tests/ -v
git add week07/lab07_contact_book.py week07/lab07_api_client.py
git commit -m "Complete Lab 07"
git push

A green Week 07 badge earns the complete 10 points.

Where these skills are used

Files and APIs use many of the same data skills

• Application integration: Payments, authentication, maps, and notifications often use APIs
• Data exchange: JSON is common in web services, configuration, and persisted application data
• Review questions: What can fail, which errors should the caller see, and what data shape should the function return?
• Project connection: A small API client demonstrates networking, parsing, and failure handling

Key takeaways

• Persistence: Files let your programs remember state between runs
• JSON: A common text format used in files and many APIs
• APIs: Let your apps communicate with other systems over the internet
• HTTP: A request-response protocol used by the web
• Requests library: Provides a Python interface for HTTP requests
• Error handling: Define behavior for HTTP, connection, timeout, and decoding failures
• The connection: json.load() (files) and response.json() (APIs) both give you Python dicts

Resources

Documentation and learning materials

• Python file I/O tutorial
• JSON module documentation
• Requests library documentation
• HTTP methods reference
• Public APIs directory
• Real Python: Working with APIs

Questions?

Next steps: - Review Lab 07 instructions in week07/lab07.md - Implement the two specified modules - Run the offline tests before pushing

Office hours: Bring the failing test, traceback, and current diff.

Wrap-up reminders: - Remind students that the tests replace network calls with local fakes - Trace one request failure and one JSON-decoding failure through get_api_data - Use the lab contract, not an optional live API, to resolve ambiguous behavior