Puzzle 2 - L3akCTF 2025

This is the second challenge in a 5-part series involving solving a scrablmed image puzzles (10 of them in this case) in record time, for the most part.

The first part is solvable manually, but this level increases the number of pieces and reduces the time limit for each puzzle, making atomation the clear path forward.

Hint Discovery

Initial analysis of the web page revealed hint. The <h1> element containing the puzzle’s title was also an <a> tag, linking to an external site (which is X, but not always). Following this link led to a Twitter post that featured the original, fully-assembled image for the current puzzle.

This oversight changed the problem entirely. Instead of a complex computer vision challenge involving pattern recognition, it became a more straightforward task: scrape the solution, map the pieces, and execute the moves. The source image URL on Twitter could even be modified from ?name=small to ?name=4096x4096 to retrieve a high-resolution version, which was ideal for creating accurate reference tiles.

Methodology

The solver was built in Python using Selenium for browser automation and OpenCV for image processing. The script follows these steps for each puzzle:

Navigate and Scrape: Use Selenium to load the puzzle page, find the Twitter link, open it in a new tab, and extract the high-resolution source image URL.
Create Reference Tiles: Download the source image. After some minor cropping and resizing to match the puzzle’s dimensions, slice it into a grid of “reference tiles” using OpenCV. This grid represents the solved state.
Extract Current State: Scrape the src attribute of each puzzle piece on the board. Since these were Base64 encoded, they were decoded in-memory into OpenCV image objects.
Map Pieces: Match each scrambled tile on the board to its corresponding reference tile.
Execute Swaps: Use Selenium’s ActionChains to perform all required swaps in bundled and more efficient sequences.

A more sophisticated method was to communicate with the backend api directly to retrieve the puzzle pieces without all the webscraping shenanigans, but I only found out about that part when moving on the next part which integrates that functionality. But, to be honest, this method is a lot cooler as you can see the puzzle pieces being rearranged in the browser in real time.

Is it less efficient? Yes, but I’d be danmed not to say it’s a hell of a lot cooler! Plus, after each puzzle is solved, we’ll be ✨ delighted ✨ with some questionable furry artwork. Seriously… WHYYYY 💀

Furry lol

Technical Implementation

Two components were key to the script’s success: the matching algorithm and the execution of swaps.

Optimal Piece Matching

To reliably match the scrambled pieces to the reference tiles, a simple 1-to-1 comparison is insufficient. This is a classic assignment problem, perfectly suited for the Hungarian algorithm, which we can access via scipy.optimize.linear_sum_assignment.

First, a cost matrix is constructed. Each cell (i, j) in the matrix represents the “cost” of matching scrambled piece i to reference piece j. This cost is calculated as 1 - similarity, where similarity is the result of a normalized template match (cv2.matchTemplate) between the two tiles.

1
2
3
4
5
6
7
8
for i in trange(n):
    for j in range(n):
        res = cv2.matchTemplate(cur_tiles_processed[i], ref_tiles_processed[j], cv2.TM_CCOEFF_NORMED)
        score = np.linalg.norm(res, 'fro') #the Frobenius norm of the cost matrix
        cost_matrix[i, j] = 1 - score

row_ind, col_ind = linear_sum_assignment(cost_matrix)
piece_locations = {ref_idx: cur_idx for cur_idx, ref_idx in zip(row_ind, col_ind)}

This function returns the optimal pairing of scrambled pieces to reference slots, providing a complete map of where each piece needs to go.

Executing the Moves

Executing swaps in one shot would be optimal, but with each swap, the DOM refreshes the ids of the tiles, making it necessary to re-query the DOM of the tiles indexes. Not only that, but it could (and did) lead to race conditions with the web interface, ruining the entire procedure.

Using the mapping from the assignment step, the script calculates swaps needed to put each piece in the correct position, going left to right, lot to bottom. Each swap (a click on the source tile, followed by a click on the destination tile) is added to a Selenium.ActionChains object and then executed immediately.

This obviously makes it slower, but at least it works.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
actions = ActionChains(driver)
    for i in range(len(piece_elements)):
        correct_piece_current_pos = piece_locations[i]
        if correct_piece_current_pos != i:
            displaced_piece_ref_idx = next(key for key, value in piece_locations.items() if value == i)

            logger.info(f"Swapping piece for slot {i} (currently at {correct_piece_current_pos}) with piece for slot {displaced_piece_ref_idx} (currently at {i}).")
            current_pieces = driver.find_elements(By.CSS_SELECTOR, '.puzzle-board .puzzle-piece')

            src_el = current_pieces[correct_piece_current_pos]
            dst_el = current_pieces[i]

            actions.click(src_el).click(dst_el).perform()
            piece_locations[i], piece_locations[displaced_piece_ref_idx] = i, correct_piece_current_pos

    logger.info("All pieces placed correctly in sequence.")

The script reliably solved each puzzle in approximately 2 minutes, from page load to completion; 1 minute to build the reference map and another one to execute the swaps.

Script

Click to expand solver code

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
import os
import time
import cv2
import numpy as np
from PIL import Image
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import logging
import requests
from io import BytesIO
import base64
from collections import Counter
from tqdm import trange
import imagehash
from scipy.optimize import linear_sum_assignment


logging.basicConfig(format='SEL> %(message)s')
logger = logging.getLogger()
logger.setLevel(logging.INFO)
PUZZLE_URL = 'http://34.55.69.223:14001/puzzle'

options = webdriver.FirefoxOptions()
driver = webdriver.Firefox(options=options)
driver.get(PUZZLE_URL)
cookie = {
    'name': 'session',
    'value': 'mxyALe2gmk2lAYwX7PjP3YBxr3VW63sCu6RLnlOYuoNF7vzAQ_ySSycSOFe5spkv2mj70iqGfc2NbcWbuG6DggbME01xAQbSGGIZ6QV9UVESSNGNRneQqaUrwyz5yNN6nI9MKSoe-xYcSEAMHu0QbUsEqMTAfETN4QVuSxbMhjI='
}
driver.add_cookie(cookie)
driver.refresh()
while True:
    driver.get(PUZZLE_URL)
    try:
        WebDriverWait(driver, 90).until(
            EC.presence_of_element_located((By.CSS_SELECTOR, ".puzzle-board:not(.hide)"))
        )
    except Exception as e:
        logger.error(f"Puzzle board not found: {e}")
        driver.quit()
        exit()
    logger.info("Puzzle board loaded successfully.")


    import re
    puzzle_board = driver.find_element(By.CSS_SELECTOR, ".puzzle-board:not(.hide)")
    style = puzzle_board.get_attribute('style')

    if style is None:
        logger.error("Style attribute not found on puzzle board element.")
        driver.quit()
        exit()

    try:
        cols_match = re.search(r'grid-template-columns: repeat\((\d+),\s*([^)]+)\)', style)
        rows_match = re.search(r'grid-template-rows: repeat\((\d+),\s*([^)]+)\)', style)

        if not cols_match or not rows_match:
            raise ValueError("Could not find grid dimensions in style attribute")

        GRID_COLS = int(cols_match.group(1))
        GRID_ROWS = int(rows_match.group(1))
        TILE_WIDTH = int(cols_match.group(2).strip()[:-2])
        TILE_HEIGHT = int(rows_match.group(2).strip()[:-2])
        logger.info(f"Grid dimensions found: {GRID_ROWS} rows, {GRID_COLS} columns, each tile {TILE_WIDTH}x{TILE_HEIGHT} pixels.")

    except (ValueError, IndexError) as e:
        logger.error(f"Could not parse grid dimensions: {e}")
        driver.quit()
        exit()


    try:
        logger.info("Finding link to the original image...")
        original_window = driver.current_window_handle

        puzzle_title_h1 = WebDriverWait(driver, 10).until(
            EC.presence_of_element_located((By.ID, "puzzle-title"))
        )
        twitter_link_element = puzzle_title_h1.find_element(By.TAG_NAME, "a")
        twitter_post_url = twitter_link_element.get_attribute('href')
        if not twitter_post_url:
            raise ValueError("Found the link element but it has no href.")
        logger.info(f"Found Twitter post URL: {twitter_post_url}")

        driver.switch_to.new_window('tab')
        driver.get(twitter_post_url)

        logger.info("On Twitter page, finding the image...")
        image_element = WebDriverWait(driver, 20).until(
            EC.presence_of_element_located((By.CSS_SELECTOR, 'img[alt="Image"]'))
        )
        FULL_IMAGE_URL = image_element.get_attribute('src')
        if not FULL_IMAGE_URL:
            raise ValueError("Found the image element but it has no src.")
        FULL_IMAGE_URL = FULL_IMAGE_URL.replace("name=small", "name=4096x4096")
        logger.info(f"Found full image URL: {FULL_IMAGE_URL}")

        driver.close()
        driver.switch_to.window(original_window)
        logger.info("Switched back to the puzzle page.")

    except Exception as e:
        logger.error(f"Failed to get the full image URL: {e}")
        driver.quit()
        exit()


    #FULL_IMAGE_URL = "https://d.furaffinity.net/art/lundi/1712867238/1712867238.lundi_tabby_comm.jpg"
    response = requests.get(FULL_IMAGE_URL)
    full_img = Image.open(BytesIO(response.content))
    full_img = cv2.cvtColor(np.array(full_img), cv2.COLOR_RGB2BGR)
    logger.info("Full image downloaded and converted to OpenCV format.")

    def preprocess_for_matching(image):
        """
        Converts an image to grayscale and applies Canny edge detection
        to prepare it for robust template matching.
        """
        gray_image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

        blurred_image = cv2.GaussianBlur(gray_image, (5, 5), 0)
        edges = cv2.Canny(blurred_image, 50, 150)

        return edges

    def sharpen_image(image):
        """Applies a sharpening kernel to the image."""
        kernel = np.array([[0, -1, 0],
                           [-1, 5, -1],
                           [0, -1, 0]])
        sharpened_image = cv2.filter2D(src=image, ddepth=-1, kernel=kernel)
        return sharpened_image

    def slice_grid(img, rows, cols):
        h, w = img.shape[:2]
        slice_h, slice_w = h // rows, w // cols
        tiles = []
        for i in range(rows):
            for j in range(cols):
                y0, y1 = i * slice_h, (i + 1) * slice_h
                x0, x1 = j * slice_w, (j + 1) * slice_w
                im_slice = img[y0:y1, x0:x1]
                tiles.append(im_slice)
        return tiles

    def get_border_color(tiles):
        """Analyzes tile images to find the most common border color."""
        border_pixels = []
        for tile in tiles:
            h, w, _ = tile.shape
            if h > 1 and w > 1:
                border_pixels.extend([tuple(p) for p in tile[0, :, :]])
                border_pixels.extend([tuple(p) for p in tile[h-1, :, :]])
                border_pixels.extend([tuple(p) for p in tile[1:h-1, 0, :]])
                border_pixels.extend([tuple(p) for p in tile[1:h-1, w-1, :]])

        if not border_pixels:
            return [0, 0, 0]

        most_common_pixel = Counter(border_pixels).most_common(1)[0][0]
        return list(most_common_pixel)

    def phash(tile):
        # convert BGR->RGB->PIL image
        pil = Image.fromarray(cv2.cvtColor(tile, cv2.COLOR_BGR2RGB))
        return imagehash.phash(pil)



    first_dim = driver.find_elements(By.CSS_SELECTOR, '.puzzle-board .puzzle-piece')
    img_url = first_dim[0].get_attribute('src')
    if not img_url:
        logger.warning("No image URL found for a puzzle piece.")
        exit()
    if img_url.startswith('data:image'):
        header, encoded = img_url.split(',', 1)
        data = base64.b64decode(encoded)
        img = Image.open(BytesIO(data))
        TILE_WIDTH = img.width
        TILE_HEIGHT = img.height
    logger.info(f"Puzzle piece dimensions: {TILE_WIDTH}x{TILE_HEIGHT} pixels.")
    border_size = 5

    initial_pieces = driver.find_elements(By.CSS_SELECTOR, '.puzzle-board .puzzle-piece')
    initial_tiles = []
    for piece_element in initial_pieces:
        img_url = piece_element.get_attribute('src')
        if not img_url:
            logger.warning("No image URL found for a puzzle piece.")
            exit()
        if img_url.startswith('data:image'):
            header, encoded = img_url.split(',', 1)
            data = base64.b64decode(encoded)
            img = Image.open(BytesIO(data))
            initial_tiles.append(cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR))

    border_color = get_border_color(initial_tiles)
    logger.info(f"Detected border color: {border_color}")
    original_width = full_img.shape[1]
    original_height = full_img.shape[0]
    logger.info(f"Original image dimensions: {original_width}x{original_height} pixels.")
    TWITTER_WIDTH = TILE_WIDTH * GRID_COLS
    TWITTER_HEIGHT = TILE_HEIGHT * GRID_ROWS
    crop_width = border_size + (original_width -TWITTER_WIDTH)//2
    crop_height = border_size + (original_height - TWITTER_HEIGHT)//2
    logger.info(f"Target Twitter image dimensions: {TWITTER_WIDTH}x{TWITTER_HEIGHT} pixels.")
    #twitter_img = cv2.resize(full_img, (TWITTER_WIDTH, TWITTER_HEIGHT), interpolation=cv2.INTER_AREA)

    h, w, _ = full_img.shape
    full_img = full_img[crop_height*2:h-crop_height*2, crop_width:w-crop_width]
    logger.info(f"Cropped {crop_height}x{crop_width}px from the sides of the full image.")


    twitter_img = cv2.copyMakeBorder(
        full_img,
        top=border_size,
        bottom=border_size,
        left=border_size,
        right=border_size,
        borderType=cv2.BORDER_CONSTANT,
        value=[int(c) for c in border_color]
    )
    logger.info(f"Twitter image dimensions after border: {twitter_img.shape[1]}x{twitter_img.shape[0]} pixels.")
    ref_tiles = slice_grid(twitter_img, GRID_ROWS, GRID_COLS)
    """
    logger.info("Sharpening reference tiles...")
    ref_tiles = [sharpen_image(tile) for tile in ref_tiles]
    """
    logger.info("Reference tiles sliced from full image.")
    edge_detect = False
    deebooged = False
    while True:
        piece_elements = driver.find_elements(By.CSS_SELECTOR, '.puzzle-board .puzzle-piece')
        cur_tiles = []
        for i, piece_element in enumerate(piece_elements):
            img_url = piece_element.get_attribute('src')
            if not img_url:
                logger.warning(f"No image URL found for puzzle piece {i}.")
                exit()
            if img_url.startswith('data:image'):
                header, encoded = img_url.split(',', 1)
                data = base64.b64decode(encoded)
                img = Image.open(BytesIO(data))
            else:
                response = requests.get(img_url)
                img = Image.open(BytesIO(response.content))

            tile_img = cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR)
            cur_tiles.append(tile_img)
        logger.info("Mapping current pieces to reference tiles...")

        ref_tiles_processed = ref_tiles#[preprocess_for_matching(tile) for tile in ref_tiles]
        cur_tiles_processed = cur_tiles #[preprocess_for_matching(tile) for tile in cur_tiles]

        n = len(cur_tiles_processed)
        cost_matrix = np.zeros((n, n))

        logger.info("Building cost matrix using template matching on edges...")
        for i in trange(n):
            for j in range(n):
                res = cv2.matchTemplate(cur_tiles_processed[i], ref_tiles_processed[j], cv2.TM_CCOEFF_NORMED)
                score = np.linalg.norm(res, 'fro')
                cost_matrix[i, j] = 1 - score

        row_ind, col_ind = linear_sum_assignment(cost_matrix)
        piece_locations = {ref_idx: cur_idx for cur_idx, ref_idx in zip(row_ind, col_ind)}

                # --- DEBUG: Save matched images for inspection ---
        if not deebooged:
            logger.info("Debugging mode: Saving matched images for inspection.")
            deebooged = True
            debug_dir = "debug_matches"
            os.makedirs(debug_dir, exist_ok=True)

            logger.info(f"Saving matched images to '{debug_dir}/' for debugging...")
            for ref_idx, cur_idx in piece_locations.items():
                ref_img_to_save = ref_tiles[ref_idx]
                cur_img_to_save = cur_tiles[cur_idx]
                h, w, _ = cur_img_to_save.shape
                if ref_img_to_save.shape[0] != h or ref_img_to_save.shape[1] != w:
                    logging.warning(f"Resizing reference image {ref_idx} to match current image dimensions.")
                    logging.warning(f"Reference image shape: {ref_img_to_save.shape}, Current image shape: {cur_img_to_save.shape}")
                    ref_img_to_save = cv2.resize(ref_img_to_save, (w, h), interpolation=cv2.INTER_AREA)
                abs_img_to_save = cv2.absdiff(ref_img_to_save, cur_img_to_save)

                combined_img = np.concatenate((ref_img_to_save, cur_img_to_save,abs_img_to_save), axis=1)

                cv2.imwrite(os.path.join(debug_dir, f"match_{ref_idx:03d}_(ref_vs_cur).png"), combined_img)
            # --- END DEBUG ---

        current_pieces = driver.find_elements(By.CSS_SELECTOR, '.puzzle-board .puzzle-piece')

        actions = ActionChains(driver)

        for i in range(len(piece_elements)):
            correct_piece_current_pos = piece_locations[i]
            if correct_piece_current_pos != i:
                displaced_piece_ref_idx = next(key for key, value in piece_locations.items() if value == i)

                logger.info(f"Swapping piece for slot {i} (currently at {correct_piece_current_pos}) with piece for slot {displaced_piece_ref_idx} (currently at {i}).")
                current_pieces = driver.find_elements(By.CSS_SELECTOR, '.puzzle-board .puzzle-piece')

                src_el = current_pieces[correct_piece_current_pos]
                dst_el = current_pieces[i]

                actions.click(src_el).click(dst_el).perform()
                piece_locations[i], piece_locations[displaced_piece_ref_idx] = i, correct_piece_current_pos

        logger.info("All pieces placed correctly in sequence.")


        try:
            WebDriverWait(driver, 10).until(
                EC.presence_of_element_located((By.XPATH, "//*[contains(text(), 'Solved')]"))
            )
            logger.info("Puzzle solved successfully!")
            break
        except Exception:
            if edge_detect:
                logger.error("Puzzle not solved even with edge detect, manually check the pieces.")
                driver.quit()
                exit()
            logger.error("Puzzle not solved. Retrying with edge detection...")
            edge_detect = True

    print('Puzzle solving attempt complete.')
    logger.info("Starting next puzzle...")