This project uses only what Parts 1 to 5 taught. The goal is a script you can read from top to bottom and trust. Raw location data is almost never clean, and the same steps appear in every GIS data job.
The problem
You receive records typed by different people. Names have odd capitals and spaces, numbers have commas, one latitude is outside the possible range, one population is missing, and one city appears twice.
# Step 1: the messy input. Each record is a dictionary, and every value is TEXT, as it would be from a CSV.
# Look for the problems: extra spaces, capitals all over, a comma inside a number,
# a latitude of 120.5 (impossible), a blank population, "abc" as a latitude, and Bengaluru twice.
raw_records = [
{"city": " bengaluru ", "state": "KARNATAKA", "population": "8,443,675", "lat": "12.9716", "lon": "77.5946"},
{"city": "MYSURU", "state": "karnataka", "population": "920550", "lat": "12.2958", "lon": "76.6394"},
{"city": "chennai", "state": "Tamil Nadu ", "population": "4646732", "lat": "13.0827", "lon": "80.2707"},
{"city": "Mumbai", "state": "maharashtra", "population": "12442373", "lat": "19.0760", "lon": "72.8777"},
{"city": "Nowhere", "state": "Karnataka", "population": "5000", "lat": "120.5", "lon": "77.0"},
{"city": "Pune", "state": "Maharashtra", "population": "", "lat": "18.5204", "lon": "73.8567"},
{"city": "Hubballi", "state": "Karnataka", "population": "943857", "lat": "abc", "lon": "75.1240"},
{"city": "Bengaluru", "state": "Karnataka", "population": "8443675", "lat": "12.9716", "lon": "77.5946"},
]
print(len(raw_records), "raw records")
8 raw records
The plan
| Step | Job | Function |
|---|---|---|
| 1 | Clean text | clean_text |
| 2 | Convert numbers, validate | to_number, clean_record |
| 3 | Add derived values | size_class |
| 4 | Drop duplicates, keep a log | clean_all |
| 5 | Write outputs | save_outputs |
Writing the plan first, even on paper, makes the code easy to follow. Each step becomes a short function.
Step 1: small helper functions
# Step 2: three small helper functions. Each does ONE job, so each is easy to test.
# Helper 1: tidy a piece of text (remove outer spaces, capitalise each word)
def clean_text(text):
return text.strip().title()
# Helper 2: turn text into a number, or give back None when that is not possible.
# kind=float means "make a decimal number" unless the caller asks for int (a whole number).
def to_number(text, kind=float):
"""Turn text such as '8,443,675' into a number, or return None."""
try:
return kind(text.replace(",", "").strip()) # remove commas and spaces, then convert
except ValueError: # "abc" and "" cannot be converted
return None
# Helper 3: label a city by its population
def size_class(population):
if population >= 5_000_000:
return "metro"
if population >= 1_000_000:
return "large"
if population >= 100_000:
return "medium"
return "small"
# Try each helper once before moving on
print(clean_text(" tamil NADU "))
print(to_number("8,443,675", int), to_number("abc"), to_number(""))
print(size_class(920550))
Tamil Nadu
8443675 None None
medium
Each helper does one thing and can be tested alone. to_number returns None when the text cannot be converted, and the caller decides what to do. An empty string raises a ValueError in float(""), so a missing population also becomes None.
Step 2: clean and validate one record
# Step 3: clean ONE record. The answer is always a pair: (the clean record, None) if it passed,
# or (None, the reason) if it must be rejected.
def clean_record(raw):
"""Return (clean_record, None) or (None, reason)."""
# 3a. use the helpers on each field
city = clean_text(raw["city"])
state = clean_text(raw["state"])
population = to_number(raw["population"], int)
lat = to_number(raw["lat"])
lon = to_number(raw["lon"])
# 3b. the checks. The first one that fails ends the function with a reason.
if lat is None or lon is None:
return None, "coordinates are not numbers"
if not (-90 <= lat <= 90 and -180 <= lon <= 180):
return None, "coordinates out of range"
if population is None:
return None, "population missing"
# 3c. every check passed, so build the clean record, adding the size label
record = {
"city": city,
"state": state,
"population": population,
"lat": lat,
"lon": lon,
"size": size_class(population),
}
return record, None
# Test it on one good record (the first) and one bad record (the fifth, with latitude 120.5)
print(clean_record(raw_records[0]))
print(clean_record(raw_records[4]))
({'city': 'Bengaluru', 'state': 'Karnataka', 'population': 8443675, 'lat': 12.9716, 'lon': 77.5946, 'size': 'metro'}, None)
(None, 'coordinates out of range')
The function returns two values: the clean record and the reason it was rejected. Exactly one of them is None. Returning a reason, instead of silently skipping, lets you report what was lost.
Step 3: process all records
# Step 4: clean ALL the records, and also remove duplicates.
def clean_all(records):
good = [] # records that passed
rejected = [] # a note about each record that did not
seen = set() # remembers which cities we have already kept
# enumerate(..., start=1) numbers the rows from 1, so the report can say "row 5"
for number, raw in enumerate(records, start=1):
record, reason = clean_record(raw) # the pair from Step 3
if record is None: # it failed a check: note why, move on to the next record
rejected.append({"row": number, "city": raw["city"].strip(), "reason": reason})
continue
# a duplicate means the same city name at the same place
key = (record["city"], record["lat"], record["lon"])
if key in seen:
rejected.append({"row": number, "city": record["city"], "reason": "duplicate"})
continue
seen.add(key) # remember it, then keep the record
good.append(record)
return good, rejected
good, rejected = clean_all(raw_records)
print(len(good), "kept,", len(rejected), "rejected")
for item in rejected:
print(item)
4 kept, 4 rejected
{'row': 5, 'city': 'Nowhere', 'reason': 'coordinates out of range'}
{'row': 6, 'city': 'Pune', 'reason': 'population missing'}
{'row': 7, 'city': 'Hubballi', 'reason': 'coordinates are not numbers'}
{'row': 8, 'city': 'Bengaluru', 'reason': 'duplicate'}
Eight records went in, four came out and four are explained in the log. The seen set remembers the city and coordinates of every kept record, and key in seen is a fast duplicate check. Row 1 and row 8 are the same city, so the second one is rejected.
Step 4: write the outputs
# Step 5: save the results as files so other programs (or people) can use them
import csv
import json
from pathlib import Path
def save_outputs(good, rejected, folder="cleaner_output"):
out = Path(folder)
out.mkdir(exist_ok=True) # make the folder if needed
# the clean cities as a CSV table
fields = ["city", "state", "population", "lat", "lon", "size"]
with open(out / "cities_clean.csv", "w", newline="", encoding="utf-8") as f:
writer = csv.DictWriter(f, fieldnames=fields)
writer.writeheader()
writer.writerows(good)
# the clean cities as JSON
with open(out / "cities_clean.json", "w", encoding="utf-8") as f:
json.dump(good, f, indent=2)
# the rejected records and their reasons, so nothing disappears silently
with open(out / "rejected.json", "w", encoding="utf-8") as f:
json.dump(rejected, f, indent=2)
return out
out = save_outputs(good, rejected)
print(sorted(p.name for p in out.iterdir())) # the three files created
print((out / "cities_clean.csv").read_text()) # show the CSV contents
['cities_clean.csv', 'cities_clean.json', 'rejected.json']
city,state,population,lat,lon,size
Bengaluru,Karnataka,8443675,12.9716,77.5946,metro
Mysuru,Karnataka,920550,12.2958,76.6394,medium
Chennai,Tamil Nadu,4646732,13.0827,80.2707,large
Mumbai,Maharashtra,12442373,19.076,72.8777,metro
The names are tidy, the numbers are numbers, and the size column was derived from population. Keeping rejected.json beside the clean file means anyone can see what was dropped and why.
Step 5: test the pipeline
Tests catch mistakes before they spread into later analysis.
# Step 6: automatic checks. Each assert must be True, otherwise Python stops and shows which one failed.
assert len(good) + len(rejected) == len(raw_records) # every record is either kept or rejected, none lost
assert all(-90 <= r["lat"] <= 90 for r in good) # no impossible latitudes in the clean data
assert all(r["population"] > 0 for r in good) # populations are positive
assert len({r["city"] for r in good}) == len(good) # no city appears twice
assert clean_record(raw_records[5])[1] == "population missing" # Pune is rejected for the right reason
print("all checks passed")
all checks passed
The first check is the most useful. Every input record must end up either kept or rejected, so nothing disappears quietly.
Step 6: hand the file to pandas and GeoPandas
The CSV is ready for the next parts. After Parts 7 and 10 this is all it takes to make a map layer:
import pandas as pd
import geopandas as gpd
df = pd.read_csv("cleaner_output/cities_clean.csv")
points = gpd.GeoDataFrame(df, geometry=gpd.points_from_xy(df["lon"], df["lat"]), crs="EPSG:4326")
Longitude is x and latitude is y, so points_from_xy takes longitude first. EPSG:4326 is the geographic coordinate system WGS 84, which is what GPS and most web data use.
Run this part as a notebook
The code on this page is also a Jupyter notebook. It has the theory in text cells, then the code, then guided practice, an independent assignment and hidden solutions, so you can try each task before you look. Download the Part 6 notebook, open it in JupyterLab and run the cells from top to bottom. Part 1 shows how to install and start Jupyter.
Extending the project
- Read the raw records from a CSV file instead of a list, using
csv.DictReader. - Add a rough check that points fall inside India's bounding box.
- Add a
testssection as functions withassertand run them at the top of the script. - Wrap steps in a
main()function and call it underif __name__ == "__main__":so the file can also be imported.
Common mistakes
- Dropping bad records without logging them.
- Cleaning text but forgetting to convert numbers.
- Checking only a few rows by eye. Add count checks that compare input and output sizes.
- Writing one very long function. Short functions are easier to test.
- Putting latitude first when creating points. The order is longitude, latitude.
Key takeaways
Next: Pandas first steps.