Challenge
The task was to extract the complete WIRTGEN, VÖGELE and HAMM catalogue from the closed WIDOS desktop application, which has no API or built-in export. More than 31,000 assemblies lived behind a server-driven tree, technical drawings were assembled from tiles, and an old embedded Chromium, an unstable remote machine and duplicate catalogues made conventional browser scraping unreliable.
Solution
I reverse-engineered the WIDOS interface and network behaviour, connected to its embedded browser through Chrome DevTools Protocol and controlled the catalogue with JavaScript injections. The extractor expands the tree, collects part records and saves progress row by row, while a dedicated module captures the dynamic drawing area through CDP and verifies each image by tile-set changes and hashes. The data is then normalised, deduplicated and loaded into a FastAPI and SQLite web catalogue.
Outcome
The resulting database covers 130 machine models, 735,906 positions, 59,254 unique part numbers, 31,131 assemblies and 29,392 PNG drawings. A total of 28,624 drawings are linked to assemblies, providing roughly 92% coverage. In the web catalogue, users can follow machine → assembly → drawing and table, or search a part number directly and open the relevant diagram and position.
