IS 4010: Application Development with Artificial Intelligence

Week 07: Working with external data

Brandon M. Greenwell

Working with external data

From local files to web APIs

  • The progression: Python values → files → JSON → HTTP responses
  • Why this matters: Applications often exchange data across process and system boundaries
  • Examples: GitHub, weather, mapping, and payment services document APIs for client applications
  • This week: Persist contacts locally and implement a testable HTTP client

Part 1: files and JSON

The problem: programs have no persistent state

Values in memory disappear when the process ends

  • Every program so far: Variables, lists, dictionaries - all gone when the program exits
  • Real apps need memory: Contact lists, game progress, user preferences, shopping carts
  • The solution: Persistence - saving data to permanent storage (files, databases)
  • This enables: Apps that remember state between runs, data analysis on large datasets, sharing data between programs

Reading and writing files in Python

The standard pattern: with open() context manager

# Write to a file (overwrites existing content)
with open("my_note.txt", "w") as f:
    f.write("Hello from our application!")

# Read the content back
with open("my_note.txt", "r") as f:
    content = f.read()
    print(content)  # Output: Hello from our application!
  • Why with?: Automatically closes the file even if errors occur
  • File modes: "r" (read), "w" (write/overwrite), "a" (append)
  • Practice: Use with so the file closes reliably when the block exits
  • Python file I/O documentation

Your turn: file writing practice

  1. Create a file called todo.txt with your top 3 tasks
  2. Read it back and print the contents
  3. Use append mode ("a") to add a 4th task
# Your code here

The problem with plain text files

Text is unstructured and hard to parse

# Saving a contact list to a text file - messy!
with open("contacts.txt", "w") as f:
    f.write("Alice|alice@example.com|555-1234\n")
    f.write("Bob|bob@example.com|555-5678\n")

# Reading it back - lots of manual parsing
with open("contacts.txt", "r") as f:
    for line in f:
        parts = line.strip().split("|")
        name, email, phone = parts
        print(f"{name}: {email}")
  • Problems: Custom delimiters (|), fragile parsing, no nested data, no type info
  • What if: Email contains |? What about optional fields? Nested addresses?
  • The real solution: We need a standard data format everyone agrees on

Enter JSON: a common data format

JavaScript Object Notation

  • JSON is a simple, human-readable text format for representing structured data
  • Created: Early 2000s by Douglas Crockford - became the web’s standard (JSON.org)
  • Why it is common: It is text-based, language-independent, and maps naturally to familiar collections
  • Ubiquity: APIs, config files, databases, logs - JSON is everywhere
  • Python advantage: JSON maps directly to Python’s built-in types

JSON structure maps to Python

Almost identical syntax

JSON Type Python Type Example
Object {} Dictionary {"name": "Alice", "age": 30}
Array [] List [1, 2, 3, 4, 5]
String String "hello"
Number int/float 42, 3.14
Boolean Boolean true → True, false → False
Null None null → None
  • Key insight: JSON objects and arrays map closely to Python dictionaries and lists
  • Limitation: JSON supports fewer data types than Python

Using Python’s json library

Two key functions: dump() and load()

import json

# Python dictionary (complex, nested structure)
contacts = {
    "people": [{"name": "Alice", "email": "alice@example.com", "age": 30},
        {"name": "Bob", "email": "bob@example.com", "age": 25}
    ],
    "count": 2
}

# Write Python object to JSON file
with open("contacts.json", "w") as f:
    json.dump(contacts, f, indent=4)  # indent=4 makes it readable

# Read JSON file back into Python object
with open("contacts.json", "r") as f:
    data = json.load(f)
    print(data["people"][0]["name"])  # Output: Alice
  • json.dump(obj, file): Python object → JSON file
  • json.load(file): JSON file → Python object
  • indent=4: Makes JSON human-readable (use in development, skip in production)
  • Python json module docs

What the generated JSON looks like

contacts.json

{
    "people": [{
            "name": "Alice",
            "email": "alice@example.com",
            "age": 30
        },
        {
            "name": "Bob",
            "email": "bob@example.com",
            "age": 25
        }
    ],
    "count": 2
}
  • Human-readable: You can open it in any text editor
  • Standard format: Any language can read this (JavaScript, Java, C#, etc.)
  • Portable: Email this file to a teammate, they can load it instantly

Your turn: JSON practice

  1. Create a Python dictionary representing your favorite movies (title, year, rating)
  2. Save it to movies.json with nice formatting
  3. Load it back and print the movie with the highest rating
# Your code here

Why JSON matters for APIs

The connection to part 2

  • Files are local: Your computer reads/writes JSON files
  • APIs are remote: Other computers send/receive JSON over the internet
  • Same format: The JSON you just learned works for BOTH
  • This means: Once you can work with JSON files, you can work with web APIs
  • Coming up: How to fetch JSON from remote servers using HTTP requests

Part 2: working with APIs

What is an API?

Application Programming Interface

  • An API is a set of rules that allows different software applications to communicate
  • Working analogy: A restaurant menu
    • Menu (API) tells you what you can order (available operations)
    • You don’t need to know how the kitchen works (implementation details)
    • You just make a request, and get food back (data)
  • APIs define: What operations are available, what data to send, what you get back
  • What is an API? (MDN)

Web APIs: applications exchanging data

HTTP as the communication protocol

  • Web APIs use HTTP (HyperText Transfer Protocol) to communicate over the internet
  • Same protocol: Your web browser uses HTTP to load websites
  • The exchange:
    1. Your app sends an HTTP request to a URL
    2. Remote server processes the request
    3. Server sends back an HTTP response with data (usually JSON!)
  • Connection: Both paths can produce Python dictionaries and lists, but HTTP adds status codes, latency, authentication, and network failures

Anatomy of a web API request

URL structure and HTTP methods

https://api.github.com/users/octocat/repos
└─┬─┘ └────────┬─────────┘ └─────┬────────┘
  │          base URL         endpoint path
protocol
  • Base URL: https://api.github.com - the server you’re talking to
  • Endpoint: /users/octocat/repos - the specific resource you want
  • HTTP method: GET (retrieve data), POST (send data), PUT (update), DELETE (remove)
  • For now: We’ll focus on GET requests to retrieve data
  • HTTP methods explained

Installing the requests library

A widely used third-party HTTP library

uv add requests
  • Not built-in: Unlike json, we need to install requests
  • Why requests?: A concise interface for common HTTP operations
  • Standard-library alternative: Python includes urllib.request; urllib3 is a separate third-party package

Making your first API request

GET request to a public API

import requests

# Make a GET request to the PokeAPI
url = "https://pokeapi.co/api/v2/pokemon/pikachu"
response = requests.get(url, timeout=10)

# Check if the request was successful
if response.status_code == 200:
    # Parse the JSON response into a Python dictionary
    data = response.json()
    print(f"Name: {data['name'].title()}")
    print(f"Height: {data['height']} decimetres")
    print(f"Weight: {data['weight']} hectograms")
else:
    print(f"Error: Received status code {response.status_code}")
  • requests.get(url): Makes HTTP GET request, returns response object
  • response.status_code: HTTP status code (200 = success)
  • response.json(): Parses JSON response → Python dict (same as json.load()!)
  • PokeAPI documentation

Your turn: explore PokeAPI

  1. Fetch data for a different Pokemon (replace “pikachu” with another name)
  2. Print the Pokemon’s types (hint: look at data['types'])
  3. Try an invalid Pokemon name - what happens?
# Your code here

HTTP status codes

The server’s way of telling you what happened

Code Meaning Example
200 OK - Success Data retrieved successfully
201 Created New resource created
400 Bad Request Invalid data sent
401 Unauthorized Need authentication
404 Not Found Resource doesn’t exist
500 Server Error Something broke on server
  • Check failures: Inspect the status or call raise_for_status() before processing data
  • Error handling: Different codes need different responses
  • HTTP status codes reference

The JSON connection: same format, different source

Comparison: files and APIs

Reading JSON from a file:

import json

with open("data.json", "r") as f:
    data = json.load(f)
    print(data["name"])

Reading JSON from an API:

import requests

response = requests.get(url)
data = response.json()
print(data["name"])
  • Same result: Both give you a Python dictionary
  • Same skills: Working with dicts, lists, accessing nested data
  • Different source: One is local, one is remote
  • Key takeaway: JSON knowledge transfers directly between files and APIs

API examples

Examples with different access requirements

Your turn: try different APIs

Pick one of the APIs above and: 1. Read its documentation to find an interesting endpoint 2. Make a GET request to that endpoint 3. Parse and display the interesting data from the response

# Your code here - explore a new API!

Error handling with APIs

Networks are unreliable - plan for failure

import requests

url = "https://api.example.com/data"

try:
    response = requests.get(url, timeout=5)  # 5 second timeout
    response.raise_for_status()  # Raises exception for 4xx/5xx codes

    data = response.json()
    print(f"Success: {data}")

except requests.exceptions.Timeout:
    print("Error: Request timed out")
except requests.exceptions.ConnectionError:
    print("Error: Could not connect to server")
except requests.exceptions.HTTPError as e:
    print(f"Error: HTTP {e.response.status_code}")
except requests.exceptions.JSONDecodeError:
    print("Error: Response was not valid JSON")
  • Timeouts: Set with timeout= parameter (seconds)
  • raise_for_status(): Converts error codes into exceptions
  • Why this matters: APIs can be down, slow, or change without notice

API practices

Being a good API citizen

  • Read the docs first: Every API has different rules and endpoints
  • Respect rate limits: Most free APIs limit requests (e.g., 1000/day)
  • Cache responses: Don’t request the same data repeatedly
  • Use timeouts: Don’t let your app hang forever waiting for response
  • Check status codes: Handle errors gracefully
  • API keys: Keep them secret (use .env files, never commit to git)
  • API development best practices

Putting it all together: files and APIs

A complete data workflow

import json
import requests

# 1. Fetch live data from API
response = requests.get(
    "https://pokeapi.co/api/v2/pokemon/ditto",
    timeout=10,
)
response.raise_for_status()
pokemon_data = response.json()

# 2. Process the data (extract what we need)
simplified = {
    "name": pokemon_data["name"],
    "height": pokemon_data["height"],
    "weight": pokemon_data["weight"],
    "types": [t["type"]["name"] for t in pokemon_data["types"]]
}

# 3. Save to local JSON file for offline use
with open("pokemon_cache.json", "w") as f:
    json.dump(simplified, f, indent=4)

print("Data fetched from API and saved locally!")
  • The pattern: Fetch → Process → Store
  • Why cache?: Faster loading, works offline, reduces API calls
  • Design question: Decide how long cached data remains acceptable and how the app behaves when a refresh fails

Your turn: complete workflow

Build a complete data workflow: 1. Choose an API from the list above 2. Fetch data from that API 3. Process/filter the data to extract interesting information 4. Save it to a JSON file 5. Load it back and display a formatted report

# Your complete workflow here

Lab 07 overview

JSON persistence and a testable API client

  • lab07_contact_book.py: Implement save_contacts_to_json and load_contacts_from_json
  • Use a context manager and indent=4; return an empty list when the file is missing
  • lab07_api_client.py: Implement get_api_data(url)
  • Use requests.get(url, timeout=10), raise_for_status(), and response.json()
  • Return None for request failures or invalid JSON
  • Do not modify the supplied tests; they replace network calls with local fakes

Where these skills are used

Files and APIs use many of the same data skills

  • Application integration: Payments, authentication, maps, and notifications often use APIs
  • Data exchange: JSON is common in web services, configuration, and persisted application data
  • Review questions: What can fail, which errors should the caller see, and what data shape should the function return?
  • Project connection: A small API client demonstrates networking, parsing, and failure handling

Key takeaways

  • Persistence: Files let your programs remember state between runs
  • JSON: A common text format used in files and many APIs
  • APIs: Let your apps communicate with other systems over the internet
  • HTTP: A request-response protocol used by the web
  • Requests library: Provides a Python interface for HTTP requests
  • Error handling: Define behavior for HTTP, connection, timeout, and decoding failures
  • The connection: json.load() (files) and response.json() (APIs) both give you Python dicts

Resources

Documentation and learning materials

Questions?

Next steps: - Review Lab 07 instructions in week07/lab07.md - Implement the two specified modules - Run the offline tests before pushing

Office hours: Bring the failing test, traceback, and current diff.