Tracking Hurricanes

A university data-management coursework: take the U.S. National Hurricane Center's raw advisory dumps (HTML built for display, not parsing) and turn them into clean, per-storm CSVs of position, pressure, and intensity over time, using nothing but bash, sed, and grep.

Bash · sed/grep · matplotlib · source

The data was never meant to be parsed

NHC advisory pages are HTML tables intended for a browser, not a downstream script: no consistent delimiter, storm name and file identifier repeated inline every five rows, units glued directly onto numbers (965mb, 85knots), and a fixed 13-line description header that has to be stripped before any of the actual data starts. There's no API here, just markup, so the whole exercise was building a pipeline that could reduce that into a table with sed and grep alone, since that was the point of the coursework.

The pipeline

The script takes an input file and output path, and runs the advisory dump through a chain of substitutions, each one removing exactly one layer of markup noise:

cat "$INPUT" | grep "tr" > tmp                                  # keep only table rows
sed '1,13d' tmp \
  | sed -E 's/^.*<tr><td>//' \
  | sed -E 's/<\/td><\/tr>.*$//' \
  | sed 's/<B>//g; s/<\/B>//g' \
  | sed -E 's/<hr>//g' \
  | sed 's/^[[:space:]]*//' \
  | sed 's/mb.*/mb/; s/knots.*/knots/' > tmp2

grep "[0-9]" tmp2 | sed '0~5d' | sed 'N;N;N;s/\n/,/g' > tmp3       # drop the repeated
                                                                    # storm-name line every
                                                                    # 5th row, then fold every
                                                                    # 4 lines into 1 CSV row

The trickiest part wasn't stripping tags: it was the row every five lines that repeats the storm's name and file ID (e.g. LO INVEST (AL212020)), which isn't a header and isn't data, just a display artifact the source page re-emits at every timestep. Getting a clean four-column-per-row CSV meant treating "delete every Nth line" as a first-class step in the pipeline, not an afterthought: sed '0~5d' does that, followed by sed 'N;N;N;s/\n/,/g' to fold each surviving group of four lines (timestamp, lat, lon, pressure/intensity) onto one CSV line.

That's the honest shape of working with sed: almost none of the individual substitutions are hard on their own, but each one only works because of an assumption about exactly what shape the previous step left the data in. Change the order, or the advisory page's format shifts by one tag, and the whole chain silently produces garbage instead of an error: there's no schema to violate, just wrong columns nobody notices until the plot looks strange.

From CSV to storm tracks

Once each storm's CSV existed, a short matplotlib script plotted pressure and wind-intensity over time per storm, and separately rendered storm tracks over the underlying lat/lon path. Nothing exotic: the interesting work was entirely upstream, in getting from "HTML built for a browser" to "four clean numeric columns" without ever writing an HTML parser.