CS 180 Project 1 –– Images of the Russian Empire: Colorizing the Prokudin-Gorskii Photo Collection

Overview

Sergei Prokudin-Gorskii was a visionary photographer from the early 20th century who pioneered color photography. He captured his images through three exposures: red, green, and blue filters. However, there was no way to print color photos at the time. This project aims to extract and align the color channels from his digitized glass plate negatives to produce a color image.

Preprocessing

After splitting the image into the three channels, I centered each one individually around a mean of 0. Centering the channels individually aids in eliminating the global brightness bias. This will ultimately help when aligning the channels as it will block brightness bias from interfering with the alignment metrics (reviewed in the next section). Additionally, I min-max normalized each channel to reduce the effect of outliers and bound the distribution between 0 and 1.

Naive Exhaustive Search

For all my implementations, I first aligned the green channel to the blue channel, then I aligned the red channel to the blue channel, and then lastly stacked the blue-aligned red channel, blue-aligned green channel, and blue channel to form a colorized image. In other words, the blue channel was my base. For my naive exhaustive search implementation, I used a double for-loop to traverse the two channels I wanted to align over a predefined search range. I computed a metric indicating how well the channels were aligned and returned the best displacement vector over the search range. I experimented with two metrics: the sum of squared differences (SSD) and normalized cross-correlation (NCC). I observed that SSD would result in better alignments on average and was faster–– I decided to stay consistent with using SSD for the rest of the project. After returning the best displacement vector, I used np.roll to align the channels as np.roll supports pixel wrapping when pushed beyond the max dimension.

Pyramid Search

As the naive exhaustive search is inefficient for large images, my next implementation utilized coarse-to-fine image pyramids for a performance boost. Alignment is first performed on a downscaled image, where fewer details make the process faster and help capture large displacements. This result is refined at each iteration, progressively scaling the image back to its original size.

Edge Alignment

The Sobel Filter is a standard method for edge detection in images. It works by computing the gradient of an image at each pixel by sliding a filter (3x3 kernel) over the picture in the x and y directions. Any magnitude of the gradient below a certain threshold is eliminated from the alignment map. The alignment map for each channel is used to guide the alignment of the image instead of the raw pixels.

Automatic Cropping

Similar to edge alignment, my automatic cropping implementation also hinges on the Sobel Filter. Once again, a map is computed by sliding the 3x3 kernel over the image in the x and y directions to calculate the magnitude of the gradient at each pixel. The first rows and columns with pixels above a certain threshold of gradient are used to crop the image.

Final Results

No Edge Alignment, No Cropping

Displacement (G, R): (112, -8), (104, 0)

Edge Alignment, No Cropping

Displacement (G, R): (48, 0), (88, 32)

Edge Alignment, Cropping

Displacement (G, R): (-16, 0), (-32, 32)

No Edge Alignment, No Cropping

Displacement (G, R): (3, 2), (6, 3)

Edge Alignment, No Cropping

Displacement (G, R): (3, 2), (7, 3)

Edge Alignment, Cropping

Displacement (G, R): (0, 2), (1, 3)

No Edge Alignment, No Cropping

Displacement (G, R): (56, 16), (112, 8)

Edge Alignment, No Cropping

Displacement (G, R): (120, 0), (120, 8)

Edge Alignment, Cropping

Displacement (G, R): (-8, 16)

No Edge Alignment, No Cropping

Displacement (G, R): (48, 0), (128, -8)

Edge Alignment, No Cropping

Displacement (G, R): (80, 32), (176, 40)

Edge Alignment, Cropping

Displacement (G, R): (-8, 24), (8, 32)

No Edge Alignment, No Cropping

Displacement (G, R): (32, -8), (144, -24)

Edge Alignment, No Cropping

Displacement (G, R): (32, -8), (136, -24)

Edge Alignment, Cropping

Displacement (G, R): (-40, -8), (0, -24)

No Edge Alignment, No Cropping

Displacement (G, R): (56, 16), (104, 8)

Edge Alignment, No Cropping

Displacement (G, R): (56, 24), (104, 40)

Edge Alignment, Cropping

Displacement (G, R): (-16, 24), (-24, 40)

No Edge Alignment, No Cropping

Displacement (G, R): (-6, 0), (9, 1)

Edge Alignment, No Cropping

Displacement (G, R): (-3, 1), (3, 2)

Edge Alignment, Cropping

Displacement (G, R): (-9, 1), (-9, 2)

No Edge Alignment, No Cropping

Displacement (G, R): (112, -8), (104, 0)

Edge Alignment, No Cropping

Displacement (G, R): (80, 0), (176, 16)

Edge Alignment, Cropping

Displacement (G, R): (0, 8), (16, 16)

No Edge Alignment, No Cropping

Displacement (G, R): (56, -8), (120, -16)

Edge Alignment, No Cropping

Displacement (G, R): (56, -8), (224, -24)

Edge Alignment, Cropping

Displacement (G, R): (-8, -8), (32, -24)

No Edge Alignment, No Cropping

Displacement (G, R): (40, 16), (88, 24)

Edge Alignment, No Cropping

Displacement (G, R): (40, 16), (80, 24)

Edge Alignment, Cropping

Displacement (G, R): (16, 16), (32, 24)

No Edge Alignment, No Cropping

Displacement (G, R): (120, 0), (120, 8)

Edge Alignment, No Cropping

Displacement (G, R): (64, 16), (128, 8)

Edge Alignment, Cropping

Displacement (G, R): (8, 16), (8, 16)

No Edge Alignment, No Cropping

Displacement (G, R): (0, 8), (104, 16)

Edge Alignment, No Cropping

Displacement (G, R): (48, 24), (104, 40)

Edge Alignment, Cropping

Displacement (G, R): (-16, 24), (-16, 40)

No Edge Alignment, No Cropping

Displacement (G, R): (0, -8), (48, -8)

Edge Alignment, No Cropping

Displacement (G, R): (24, 0), (56, -8)

Edge Alignment, Cropping

Displacement (G, R): (-8, 0), (-8, -8)

No Edge Alignment, No Cropping

Displacement (G, R): (1, -1), (7, -1)

Edge Alignment, No Cropping

Displacement (G, R): (48, 24), (104, 40)

Edge Alignment, Cropping

Displacement (G, R): (0, 2), (2, 3)

No Edge Alignment, No Cropping

Displacement (G, R): (-8, 0), (96, 0)

Edge Alignment, No Cropping

Displacement (G, R): (32, 0), (80, 8)

Edge Alignment, Cropping

Displacement (G, R): (-24, 0), (-40, 8)

No Edge Alignment, No Cropping

Displacement (G, R): (48, 0), (96, -8)

Edge Alignment, No Cropping

Displacement (G, R): (48, 16), (96, 24)

Edge Alignment, Cropping

Displacement (G, R): (-16, 16), (-24, 24)

No Edge Alignment, No Cropping

Displacement (G, R): (40, 0), (96, 8)

Edge Alignment, No Cropping

Displacement (G, R): (32, 0), (96, 8)

Edge Alignment, Cropping

Displacement (G, R): (32, 0), (96, 8)

No Edge Alignment, No Cropping

Displacement (G, R): (128, -8), (104, -16)

Edge Alignment, No Cropping

Displacement (G, R): (32, -8), (104, -16)

Edge Alignment, Cropping

Displacement (G, R): (-32, -8), (-16, -16)