<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en">
    <title>ljufa.iz.rs</title>
    <subtitle>Personal blog of Dragan Ljubojevic</subtitle>
    <link rel="self" type="application/atom+xml" href="https://ljufa.iz.rs/atom.xml"/>
    <link rel="alternate" type="text/html" href="https://ljufa.iz.rs"/>
    <generator uri="https://www.getzola.org/">Zola</generator>
    <updated>2026-10-15T00:00:00+00:00</updated>
    <id>https://ljufa.iz.rs/atom.xml</id>
    <entry xml:lang="en">
        <title>Leaving Google Photos: Self-Hosted Immich on a Home NAS, Public Through a VPS Tunnel</title>
        <published>2026-10-15T00:00:00+00:00</published>
        <updated>2026-10-15T00:00:00+00:00</updated>
        
        <author>
          <name>Unknown</name>
        </author>
        
        <link rel="alternate" type="text/html" href="https://ljufa.iz.rs/posts/google-photos-to-immich/"/>
        <id>https://ljufa.iz.rs/posts/google-photos-to-immich/</id>
        
        <content type="html" xml:base="https://ljufa.iz.rs/posts/google-photos-to-immich/">&lt;p&gt;My wife and I had two Google Photos accounts, a NAS full of older photos synced from our phones with Syncthing, and a growing Google One bill. This post is about getting everything into one self-hosted &lt;a rel=&quot;external&quot; href=&quot;https://immich.app&quot;&gt;Immich&lt;/a&gt; instance: the setup, the import, and the many small traps along the way.&lt;/p&gt;
&lt;p&gt;The short version: &lt;strong&gt;the hard part is not installing Immich, it is importing years of overlapping photos without creating duplicates.&lt;/strong&gt;&lt;/p&gt;
&lt;h2 id=&quot;the-setup&quot;&gt;The Setup&lt;/h2&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color: #CDD6F4; background-color: #1E1E2E;&quot; &gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;phone (Immich app) ──HTTPS──&amp;gt; VPS: Traefik ──&amp;gt; socat relay&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;                                                  │&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;                                       WireGuard 10.13.13.0/24&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;                                                  │&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;                         home NAS: WireGuard client + socat relay ──&amp;gt; Immich :2283&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;                                                  │&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;                                 /data/PHOTO/SORTED/YYYY/MM  (read-only external library)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;                                 /data/PHOTO/IMMICH          (uploads from phones)&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;NAS&lt;/strong&gt; (an old, slow box): Immich server, Postgres and Redis in Docker. The existing photo archive (&lt;code&gt;SORTED/YYYY/MM&lt;/code&gt;) is mounted &lt;strong&gt;read-only&lt;/strong&gt; and added as an Immich &lt;em&gt;External Library&lt;/em&gt;. Phone uploads go to a separate folder, never inside the external library (otherwise every upload gets imported twice).&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Workstation (temporary, only for the initial import)&lt;/strong&gt;: the first scan of tens of thousands of photos (thumbnails, metadata, face detection, smart search, transcoding) would take days on the NAS CPU. For that period only, a second &lt;code&gt;immich-server&lt;/code&gt; container with &lt;code&gt;IMMICH_WORKERS_INCLUDE=microservices&lt;/code&gt;, plus machine learning, runs on my workstation. It connects to the NAS database and Redis and mounts the same NFS paths at the &lt;strong&gt;same absolute paths&lt;/strong&gt;, because the database stores absolute paths. Once the backlog is done, it is switched off and the NAS handles the trickle of new phone uploads on its own.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;VPS&lt;/strong&gt;: Traefik terminates TLS for the public photos hostname and relays the traffic through a WireGuard tunnel to the NAS.&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id=&quot;why-a-tunnel-instead-of-port-forwarding&quot;&gt;Why a tunnel instead of port forwarding&lt;/h3&gt;
&lt;p&gt;The NAS &lt;strong&gt;dials out&lt;/strong&gt; to the VPS and keeps the tunnel open (&lt;code&gt;PersistentKeepalive&lt;/code&gt;). Nothing at home listens on the internet, the router has no forwarded ports, and it works behind CGNAT.&lt;/p&gt;
&lt;p&gt;Both ends use &lt;code&gt;linuxserver/wireguard&lt;/code&gt;, and each published service is just a tiny &lt;code&gt;socat&lt;/code&gt; relay that shares the WireGuard container&#39;s network namespace. On the VPS:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color: #CDD6F4; background-color: #1E1E2E;&quot; &gt;&lt;code data-lang=&quot;yaml&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: #89B4FA;&quot;&gt;wireguard&lt;/span&gt;&lt;span style=&quot;color: #94E2D5;&quot;&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: #89B4FA;&quot;&gt;  image&lt;/span&gt;&lt;span style=&quot;color: #94E2D5;&quot;&gt;:&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt; lscr.io/linuxserver/wireguard&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: #89B4FA;&quot;&gt;  environment&lt;/span&gt;&lt;span style=&quot;color: #94E2D5;&quot;&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;    -&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt; PEERS=nas&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;    -&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt; SERVERURL=vps.example.com&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;    -&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt; ALLOWEDIPS=10.13.13.0/24&lt;/span&gt;&lt;span style=&quot;color: #9399B2;font-style: italic;&quot;&gt;     # only the tunnel subnet, never 0.0.0.0/0&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;    -&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt; PERSISTENTKEEPALIVE_PEERS=all&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: #89B4FA;&quot;&gt;  sysctls&lt;/span&gt;&lt;span style=&quot;color: #94E2D5;&quot;&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;    -&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt; net.ipv4.ip_forward=0&lt;/span&gt;&lt;span style=&quot;color: #9399B2;font-style: italic;&quot;&gt;          # relays end each connection; no routing into either network&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: #89B4FA;&quot;&gt;  ports&lt;/span&gt;&lt;span style=&quot;color: #94E2D5;&quot;&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;    -&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt; &amp;#39;51820:51820/udp&amp;#39;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: #89B4FA;&quot;&gt;  labels&lt;/span&gt;&lt;span style=&quot;color: #94E2D5;&quot;&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;    -&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt; &amp;quot;traefik.http.routers.photo_https.rule=Host(`photos.example.com`)&amp;quot;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;    -&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt; &amp;quot;traefik.http.services.photo.loadbalancer.server.port=2283&amp;quot;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: #89B4FA;&quot;&gt;photo&lt;/span&gt;&lt;span style=&quot;color: #94E2D5;&quot;&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: #89B4FA;&quot;&gt;  image&lt;/span&gt;&lt;span style=&quot;color: #94E2D5;&quot;&gt;:&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt; alpine/socat&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: #89B4FA;&quot;&gt;  network_mode&lt;/span&gt;&lt;span style=&quot;color: #94E2D5;&quot;&gt;:&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt; service:wireguard&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: #89B4FA;&quot;&gt;  command&lt;/span&gt;&lt;span style=&quot;color: #94E2D5;&quot;&gt;:&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt; [&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt;&amp;quot;TCP-LISTEN:2283,fork,reuseaddr&amp;quot;&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt; &amp;quot;TCP:10.13.13.2:2283&amp;quot;&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;]&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;At home, the mirror image: a WireGuard client with the peer config generated on the VPS, and a relay from the tunnel to the port Immich publishes on the host:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color: #CDD6F4; background-color: #1E1E2E;&quot; &gt;&lt;code data-lang=&quot;yaml&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: #89B4FA;&quot;&gt;photo&lt;/span&gt;&lt;span style=&quot;color: #94E2D5;&quot;&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: #89B4FA;&quot;&gt;  image&lt;/span&gt;&lt;span style=&quot;color: #94E2D5;&quot;&gt;:&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt; alpine/socat&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: #89B4FA;&quot;&gt;  network_mode&lt;/span&gt;&lt;span style=&quot;color: #94E2D5;&quot;&gt;:&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt; service:wireguard&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: #89B4FA;&quot;&gt;  command&lt;/span&gt;&lt;span style=&quot;color: #94E2D5;&quot;&gt;:&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt; [&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt;&amp;quot;TCP-LISTEN:2283,fork,reuseaddr&amp;quot;&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt; &amp;quot;TCP:host.docker.internal:2283&amp;quot;&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;]&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The relays only move bytes; Immich does its own authentication. On the VPS, Traefik adds rate limits: 10 requests per minute per IP on &lt;code&gt;/api/auth/login&lt;/code&gt; (Immich has no login lockout of its own), and a loose per-IP limit on the whole site.&lt;/p&gt;
&lt;p&gt;Two lessons from running this:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Restart the whole stack, never just &lt;code&gt;wireguard&lt;/code&gt;.&lt;/strong&gt; The relays share its network namespace. Restart only the WireGuard container and the relays stay attached to the old, dead namespace: the tunnel is up, but every request ends in a 502.&lt;/li&gt;
&lt;li&gt;If the NAS runs a VPN client, the WireGuard packets to the VPS need a policy routing rule to bypass it.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&quot;accounts-decide-before-you-import&quot;&gt;Accounts: Decide Before You Import&lt;/h2&gt;
&lt;p&gt;Immich&#39;s duplicate detection works &lt;strong&gt;per user&lt;/strong&gt;. The same photo in two accounts is stored twice and nobody warns you. This drives everything else:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;One &quot;Shared&quot; account owns the external library&lt;/strong&gt; (the family archive on the NAS). Everyone sees it through &lt;em&gt;Partner Sharing&lt;/em&gt; or shared albums.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Each person has a personal account&lt;/strong&gt; for phone backup and private stuff (scanned documents, receipts, the electronics parts I photograph for my DIY projects).&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Two things I learned the hard way:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;An external library&#39;s owner cannot be changed later.&lt;/strong&gt; I had attached the library to my first (admin) account. Instead of recreating the library, which would have meant new asset IDs, lost albums, lost face names and days of reprocessing, I simply renamed that account to &quot;Shared&quot; and created a new personal admin account.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Logging the phone app into a different account re-uploads the whole camera roll.&lt;/strong&gt; After switching my phone to the personal account, I removed the old copies from the Shared account by comparing Immich&#39;s SHA-1 checksums between the two accounts (more on that below).&lt;/li&gt;
&lt;/ol&gt;
&lt;h2 id=&quot;getting-the-data-out-google-takeout&quot;&gt;Getting the Data Out: Google Takeout&lt;/h2&gt;
&lt;p&gt;Takeout gives you a pile of independent zip files (not a split archive), 2 to 10 GB each:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color: #CDD6F4; background-color: #1E1E2E;&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: #CBA6F7;&quot;&gt;for&lt;/span&gt;&lt;span&gt; f&lt;/span&gt;&lt;span style=&quot;color: #CBA6F7;&quot;&gt; in&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt; ~/Downloads/takeout*.zip&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;;&lt;/span&gt;&lt;span style=&quot;color: #CBA6F7;&quot;&gt; do&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: #F38BA8;font-style: italic;&quot;&gt;  echo&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt; &amp;quot;&lt;/span&gt;&lt;span&gt;$f&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt;&amp;quot;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: #89B4FA;font-style: italic;&quot;&gt;  unzip&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt; -o -q &amp;quot;&lt;/span&gt;&lt;span&gt;$f&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt;&amp;quot; -d .&lt;/span&gt;&lt;span style=&quot;color: #94E2D5;&quot;&gt; ||&lt;/span&gt;&lt;span style=&quot;color: #F38BA8;font-style: italic;&quot;&gt; echo&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt; &amp;quot;FAILED: &lt;/span&gt;&lt;span&gt;$f&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span style=&quot;color: #94E2D5;&quot;&gt; &amp;gt;&amp;gt;&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt; unpack_failures.log&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: #CBA6F7;&quot;&gt;done&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;strong&gt;Check the numbering first.&lt;/strong&gt; One of our exports was missing two parts out of 96 and nobody noticed for a while.&lt;/p&gt;
&lt;p&gt;What is inside:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;Takeout/Google Photos/Photos from 2019/&lt;/code&gt; and so on: &lt;strong&gt;the actual library&lt;/strong&gt;, one folder per year.&lt;/li&gt;
&lt;li&gt;One folder per album. These contain &lt;strong&gt;copies&lt;/strong&gt; of photos from the year folders, not different photos.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;*.supplemental-metadata.json&lt;/code&gt; next to every file: capture time (UTC), GPS, description. Google often does &lt;strong&gt;not&lt;/strong&gt; write this back into the image&#39;s EXIF.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;*-edited.jpg&lt;/code&gt;: Google&#39;s edited version next to the original.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;.MP&lt;/code&gt; files: the video half of Pixel motion photos.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;Trash/&lt;/code&gt;, &lt;code&gt;Archive/&lt;/code&gt; and a lot of Google-generated &quot;memories&quot; albums (&quot;Friday afternoon in Berlin&quot;, &quot;5 years ago&quot;).&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;strong&gt;Folder names follow the account language.&lt;/strong&gt; My wife&#39;s account is in Serbian, so it was &lt;code&gt;Google слике/Слике из 2019./&lt;/code&gt; with &lt;code&gt;Отпад&lt;/code&gt; instead of &lt;code&gt;Trash&lt;/code&gt; and &lt;code&gt;-измењено&lt;/code&gt; instead of &lt;code&gt;-edited&lt;/code&gt;. Any script that hard-codes the English names silently does the wrong thing.&lt;/p&gt;
&lt;h2 id=&quot;why-i-did-not-just-run-immich-go&quot;&gt;Why I Did Not Just Run immich-go&lt;/h2&gt;
&lt;p&gt;&lt;a rel=&quot;external&quot; href=&quot;https://github.com/simulot/immich-go&quot;&gt;immich-go&lt;/a&gt; is the standard tool for importing Takeout into Immich, and it is good. I ran it with &lt;code&gt;--dry-run&lt;/code&gt; on &lt;strong&gt;one small album first&lt;/strong&gt;, and that saved me:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;For a month whose photos were all already on the NAS, it wanted to upload &lt;strong&gt;336 of 402 as new&lt;/strong&gt; and to &lt;em&gt;&quot;upgrade&quot;&lt;/em&gt; the other 66.&lt;/li&gt;
&lt;li&gt;Comparing by hand showed why: her Google library was in &lt;strong&gt;&quot;Storage saver&quot;&lt;/strong&gt; quality, so the Takeout copies were 1.2 MB recompressions of 6 MB originals. Different bytes, so Immich&#39;s checksum dedup cannot match them, and immich-go&#39;s name+date matching failed on details like &lt;code&gt;DSC_0308.JPG&lt;/code&gt; versus &lt;code&gt;DSC_0308.jpg&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;The &quot;upgrade&quot; would have &lt;strong&gt;replaced the NAS originals in Immich with Google&#39;s smaller copies.&lt;/strong&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;strong&gt;Always dry-run on a small sample that you know overlaps with what is already on the server.&lt;/strong&gt; A sample of only new photos (my first test) looks perfect and proves nothing.&lt;/p&gt;
&lt;h2 id=&quot;matching-locally-metadata-first-then-image-content&quot;&gt;Matching Locally: Metadata First, Then Image Content&lt;/h2&gt;
&lt;p&gt;So I wrote a small matcher that runs locally. No photos leave the network. The slow part is reading metadata over NFS (about an hour for 52,000 files on the NAS, ten minutes for a local Takeout); the matching itself takes minutes.&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Scan both sides once&lt;/strong&gt; into JSON lines: EXIF &lt;code&gt;DateTimeOriginal&lt;/code&gt;, camera model, dimensions, size; for videos &lt;code&gt;ffprobe&lt;/code&gt; creation time and duration; for Takeout files the time from Google&#39;s JSON sidecar.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Same capture time (to the second) + same file name&lt;/strong&gt; (case-insensitive, ignoring &lt;code&gt;-edited&lt;/code&gt; and &lt;code&gt;(1)&lt;/code&gt; suffixes): already there.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Same capture time but different name, or no EXIF at all&lt;/strong&gt;: compare the actual pixels with a &lt;strong&gt;perceptual hash&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Videos&lt;/strong&gt;: recording time ± 2 s and duration ± 1 s.&lt;/li&gt;
&lt;li&gt;Everything else is new.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;A perceptual hash is a 64-bit fingerprint of what an image &lt;em&gt;looks like&lt;/em&gt;. It survives recompression and resizing, which is exactly what Google&#39;s Storage saver does:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color: #CDD6F4; background-color: #1E1E2E;&quot; &gt;&lt;code data-lang=&quot;python&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: #CBA6F7;&quot;&gt;from&lt;/span&gt;&lt;span&gt; PIL&lt;/span&gt;&lt;span style=&quot;color: #CBA6F7;&quot;&gt; import&lt;/span&gt;&lt;span&gt; Image&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;,&lt;/span&gt;&lt;span&gt; ImageOps&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: #CBA6F7;&quot;&gt;import&lt;/span&gt;&lt;span&gt; imagehash&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;,&lt;/span&gt;&lt;span&gt; pillow_heif&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;pillow_heif&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;.&lt;/span&gt;&lt;span style=&quot;color: #89B4FA;&quot;&gt;register_heif_opener&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;()&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: #CBA6F7;&quot;&gt;def&lt;/span&gt;&lt;span style=&quot;color: #89B4FA;font-style: italic;&quot;&gt; phash&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color: #EBA0AC;font-style: italic;&quot;&gt;path&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: #CBA6F7;&quot;&gt;    with&lt;/span&gt;&lt;span&gt; Image&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;.&lt;/span&gt;&lt;span style=&quot;color: #89B4FA;&quot;&gt;open&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;(&lt;/span&gt;&lt;span&gt;path&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color: #CBA6F7;&quot;&gt; as&lt;/span&gt;&lt;span&gt; im&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;        im&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;.&lt;/span&gt;&lt;span style=&quot;color: #89B4FA;&quot;&gt;draft&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt;&amp;quot;RGB&amp;quot;&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;, (&lt;/span&gt;&lt;span style=&quot;color: #FAB387;&quot;&gt;256&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color: #FAB387;&quot;&gt; 256&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;))&lt;/span&gt;&lt;span style=&quot;color: #9399B2;font-style: italic;&quot;&gt;          # JPEG: decode at 1/8 size, much faster over NFS&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;        im&lt;/span&gt;&lt;span style=&quot;color: #94E2D5;&quot;&gt; =&lt;/span&gt;&lt;span&gt; ImageOps&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;.&lt;/span&gt;&lt;span style=&quot;color: #89B4FA;&quot;&gt;exif_transpose&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;(&lt;/span&gt;&lt;span&gt;im&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;).&lt;/span&gt;&lt;span style=&quot;color: #89B4FA;&quot;&gt;convert&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt;&amp;quot;RGB&amp;quot;&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: #CBA6F7;&quot;&gt;        return&lt;/span&gt;&lt;span&gt; imagehash&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;.&lt;/span&gt;&lt;span style=&quot;color: #89B4FA;&quot;&gt;phash&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;(&lt;/span&gt;&lt;span&gt;im&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;same&lt;/span&gt;&lt;span style=&quot;color: #94E2D5;&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color: #89B4FA;&quot;&gt; phash&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;(&lt;/span&gt;&lt;span&gt;google_copy&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color: #94E2D5;&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color: #89B4FA;&quot;&gt; phash&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;(&lt;/span&gt;&lt;span&gt;nas_original&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color: #94E2D5;&quot;&gt; &amp;lt;=&lt;/span&gt;&lt;span style=&quot;color: #FAB387;&quot;&gt; 10&lt;/span&gt;&lt;span style=&quot;color: #9399B2;font-style: italic;&quot;&gt;   # Hamming distance of 64 bits&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Only the uncertain cases get hashed, and only against candidates with a nearby capture time. Of about 5,900 content matches in my wife&#39;s library, 5,641 had distance 0, and only 16 were near the threshold.&lt;/p&gt;
&lt;p&gt;Sanity checks that caught real problems:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;About 1,100 &quot;new&quot; &lt;code&gt;DSC_*&lt;/code&gt; photos had a same-named file on the NAS. Spot checks showed different photos: the camera&#39;s counter had wrapped around. Name alone is not identity.&lt;/li&gt;
&lt;li&gt;A camera with a dead clock produced a batch of photos and videos dated &lt;strong&gt;1980&lt;/strong&gt;.&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id=&quot;the-results&quot;&gt;The Results&lt;/h3&gt;
&lt;table&gt;&lt;thead&gt;&lt;tr&gt;&lt;th&gt;&lt;/th&gt;&lt;th&gt;My wife&#39;s Takeout&lt;/th&gt;&lt;th&gt;My Takeout&lt;/th&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;
&lt;tr&gt;&lt;td&gt;Unique assets (album copies collapsed, trash excluded)&lt;/td&gt;&lt;td&gt;25,813&lt;/td&gt;&lt;td&gt;45,061&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;Already on the NAS&lt;/td&gt;&lt;td&gt;~14,900&lt;/td&gt;&lt;td&gt;~42,400&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;Really new&lt;/td&gt;&lt;td&gt;~10,500&lt;/td&gt;&lt;td&gt;~2,500&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;p&gt;Before copying, I also filtered out things that do not belong in a family archive: &lt;code&gt;FB_IMG_*&lt;/code&gt; (images saved from Facebook), screenshots, WhatsApp/Viber/Messenger images, and Google&#39;s auto-generated animations and collages. I missed some of these in the first pass and had to delete 228 files afterwards, so put the filter in from the start.&lt;/p&gt;
&lt;h2 id=&quot;copying-into-the-archive-and-an-nfs-surprise&quot;&gt;Copying Into the Archive, and an NFS Surprise&lt;/h2&gt;
&lt;p&gt;New files go into &lt;code&gt;SORTED/YYYY/MM/&lt;/code&gt; like the rest of the archive, named after the original, with the file time set to the capture time (Immich falls back to it for files without EXIF).&lt;/p&gt;
&lt;p&gt;Then the timeline showed hundreds of photos dated &lt;strong&gt;today&lt;/strong&gt;. Every copied file had the copy time, even though the script had called &lt;code&gt;os.utime()&lt;/code&gt; and a check right afterwards showed the correct date. The NFS client had cached the attributes; when the dirty data was flushed to the server later, the server stamped its own modification time over mine.&lt;/p&gt;
&lt;p&gt;The fix: flush before setting the time, re-apply at the end, and &lt;strong&gt;verify through a second mount&lt;/strong&gt; of the same export so the client cache cannot lie to you:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color: #CDD6F4; background-color: #1E1E2E;&quot; &gt;&lt;code data-lang=&quot;python&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;shutil&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;.&lt;/span&gt;&lt;span style=&quot;color: #89B4FA;&quot;&gt;copyfile&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;(&lt;/span&gt;&lt;span&gt;src&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;,&lt;/span&gt;&lt;span&gt; tmp&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;fd&lt;/span&gt;&lt;span style=&quot;color: #94E2D5;&quot;&gt; =&lt;/span&gt;&lt;span&gt; os&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;.&lt;/span&gt;&lt;span style=&quot;color: #89B4FA;&quot;&gt;open&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;(&lt;/span&gt;&lt;span&gt;tmp&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;,&lt;/span&gt;&lt;span&gt; os&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;.&lt;/span&gt;&lt;span&gt;O_RDONLY&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;)&lt;/span&gt;&lt;span&gt;; os&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;.&lt;/span&gt;&lt;span style=&quot;color: #89B4FA;&quot;&gt;fsync&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;(&lt;/span&gt;&lt;span&gt;fd&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;)&lt;/span&gt;&lt;span&gt;; os&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;.&lt;/span&gt;&lt;span style=&quot;color: #89B4FA;&quot;&gt;close&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;(&lt;/span&gt;&lt;span&gt;fd&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color: #9399B2;font-style: italic;&quot;&gt;   # push the data to the server first&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;os&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;.&lt;/span&gt;&lt;span style=&quot;color: #89B4FA;&quot;&gt;utime&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;(&lt;/span&gt;&lt;span&gt;tmp&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;, (&lt;/span&gt;&lt;span&gt;capture_time&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;,&lt;/span&gt;&lt;span&gt; capture_time&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;))&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;os&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;.&lt;/span&gt;&lt;span style=&quot;color: #89B4FA;&quot;&gt;rename&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;(&lt;/span&gt;&lt;span&gt;tmp&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;,&lt;/span&gt;&lt;span&gt; dst&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: #9399B2;font-style: italic;&quot;&gt;# ... after all files: wait a minute, re-check via the other mount point, fix what is still wrong&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Even with the &lt;code&gt;fsync&lt;/code&gt;, a later batch still had 332 of 351 copies overwritten, so the final verify-and-reapply pass is not optional. Moves within the same export (&lt;code&gt;rename&lt;/code&gt;) are not affected: they never touch the file&#39;s data.&lt;/p&gt;
&lt;h2 id=&quot;recreating-albums&quot;&gt;Recreating Albums&lt;/h2&gt;
&lt;p&gt;Copying files into an external library loses Google&#39;s albums. I recreated the meaningful ones (people, trips, events) through the Immich API and skipped the &quot;memories&quot; noise. For each album, every member is either a newly copied file or the NAS original it was matched to, so I can look up its Immich asset ID by path:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color: #CDD6F4; background-color: #1E1E2E;&quot; &gt;&lt;code data-lang=&quot;python&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: #89B4FA;&quot;&gt;api&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt;&amp;quot;POST&amp;quot;&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt; &amp;quot;/albums&amp;quot;&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;, {&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt;&amp;quot;albumName&amp;quot;&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;:&lt;/span&gt;&lt;span&gt; name&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt; &amp;quot;assetIds&amp;quot;&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;:&lt;/span&gt;&lt;span&gt; ids&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;})&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: #89B4FA;&quot;&gt;api&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt;&amp;quot;PUT&amp;quot;&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt; &amp;quot;/assets&amp;quot;&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;, {&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt;&amp;quot;ids&amp;quot;&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;:&lt;/span&gt;&lt;span&gt; favourite_ids&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt; &amp;quot;isFavorite&amp;quot;&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;:&lt;/span&gt;&lt;span style=&quot;color: #FAB387;&quot;&gt; True&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;})&lt;/span&gt;&lt;span style=&quot;color: #9399B2;font-style: italic;&quot;&gt;   # Google&amp;#39;s favourites&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;h2 id=&quot;personal-photos-out-of-the-shared-library&quot;&gt;Personal Photos Out of the Shared Library&lt;/h2&gt;
&lt;p&gt;My Takeout had albums of scanned documents, contracts and electronics parts. They had also reached the NAS through Syncthing years ago, so the whole family could see them. I:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;uploaded them to my personal account through the API (&lt;code&gt;POST /api/assets&lt;/code&gt;), using the &lt;strong&gt;NAS originals&lt;/strong&gt; where they existed, because they are better than Google&#39;s copies,&lt;/li&gt;
&lt;li&gt;checked that every one of them arrived (status &lt;code&gt;created&lt;/code&gt;, then SHA-1 against Immich&#39;s stored checksums),&lt;/li&gt;
&lt;li&gt;only then moved them out of &lt;code&gt;SORTED&lt;/code&gt; into a holding folder, and deleted that folder after another verification.&lt;/li&gt;
&lt;/ol&gt;
&lt;h2 id=&quot;switching-the-phone-to-the-personal-account&quot;&gt;Switching the Phone to the Personal Account&lt;/h2&gt;
&lt;p&gt;Immich stores a SHA-1 checksum of every asset. The phone uploads byte-identical files, so deduplicating across accounts is exact, not a guess:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color: #CDD6F4; background-color: #1E1E2E;&quot; &gt;&lt;code data-lang=&quot;python&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;shared&lt;/span&gt;&lt;span style=&quot;color: #94E2D5;&quot;&gt;   =&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt; [&lt;/span&gt;&lt;span&gt;a&lt;/span&gt;&lt;span style=&quot;color: #CBA6F7;&quot;&gt; for&lt;/span&gt;&lt;span&gt; a&lt;/span&gt;&lt;span style=&quot;color: #CBA6F7;&quot;&gt; in&lt;/span&gt;&lt;span style=&quot;color: #89B4FA;&quot;&gt; all_assets&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;(&lt;/span&gt;&lt;span&gt;SHARED_KEY&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color: #CBA6F7;&quot;&gt; if not&lt;/span&gt;&lt;span&gt; a&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;.&lt;/span&gt;&lt;span style=&quot;color: #89B4FA;&quot;&gt;get&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt;&amp;quot;libraryId&amp;quot;&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;)]&lt;/span&gt;&lt;span style=&quot;color: #9399B2;font-style: italic;&quot;&gt;   # uploads only, not the NAS library&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;personal&lt;/span&gt;&lt;span style=&quot;color: #94E2D5;&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt; {&lt;/span&gt;&lt;span style=&quot;color: #EBA0AC;font-style: italic;&quot;&gt;a&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;[&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;font-style: italic;&quot;&gt;checksum&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;]&lt;/span&gt;&lt;span style=&quot;color: #CBA6F7;&quot;&gt; for&lt;/span&gt;&lt;span&gt; a&lt;/span&gt;&lt;span style=&quot;color: #CBA6F7;&quot;&gt; in&lt;/span&gt;&lt;span style=&quot;color: #89B4FA;&quot;&gt; all_assets&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;(&lt;/span&gt;&lt;span&gt;PERSONAL_KEY&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;)}&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;dups&lt;/span&gt;&lt;span style=&quot;color: #94E2D5;&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt; [&lt;/span&gt;&lt;span style=&quot;color: #EBA0AC;font-style: italic;&quot;&gt;a&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;[&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;font-style: italic;&quot;&gt;id&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;]&lt;/span&gt;&lt;span style=&quot;color: #CBA6F7;&quot;&gt; for&lt;/span&gt;&lt;span&gt; a&lt;/span&gt;&lt;span style=&quot;color: #CBA6F7;&quot;&gt; in&lt;/span&gt;&lt;span&gt; shared&lt;/span&gt;&lt;span style=&quot;color: #CBA6F7;&quot;&gt; if&lt;/span&gt;&lt;span style=&quot;color: #EBA0AC;font-style: italic;&quot;&gt; a&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;[&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;font-style: italic;&quot;&gt;checksum&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt;&amp;quot;&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;]&lt;/span&gt;&lt;span style=&quot;color: #CBA6F7;&quot;&gt; in&lt;/span&gt;&lt;span&gt; personal&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;]&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: #89B4FA;&quot;&gt;api&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;(&lt;/span&gt;&lt;span&gt;SHARED_KEY&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt; &amp;quot;DELETE&amp;quot;&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt; &amp;quot;/assets&amp;quot;&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;, {&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt;&amp;quot;ids&amp;quot;&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;:&lt;/span&gt;&lt;span&gt; dups&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt; &amp;quot;force&amp;quot;&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;:&lt;/span&gt;&lt;span style=&quot;color: #FAB387;&quot;&gt; False&lt;/span&gt;&lt;span style=&quot;color: #9399B2;&quot;&gt;})&lt;/span&gt;&lt;span style=&quot;color: #9399B2;font-style: italic;&quot;&gt;        # to trash, restorable&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;1,406 of 1,453 old uploads matched and went to the trash. The other 47 stayed.&lt;/p&gt;
&lt;h2 id=&quot;cleaning-up-old-folders-on-the-nas&quot;&gt;Cleaning Up Old Folders on the NAS&lt;/h2&gt;
&lt;p&gt;The NAS also had a separate &lt;code&gt;VIDEO/&lt;/code&gt; tree and an &lt;code&gt;UPLOAD/&lt;/code&gt; folder with Syncthing targets from several old phones. The same matcher, pointed at those folders, found 1,822 files missing from the archive and about 3,000 duplicates (including 1,118 copies of each other). Two warnings:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Do not move files out of an active Syncthing folder.&lt;/strong&gt; With a send-receive share, a move is a delete, and Syncthing deletes the photo on the phone too. I copied from those folders instead, and stopped the shares before deleting anything.&lt;/li&gt;
&lt;li&gt;Before deleting a folder, check that &lt;strong&gt;every file in it&lt;/strong&gt; has a safe counterpart, following duplicate chains to the end. &quot;A copy of a copy&quot; is only safe if the last copy survives.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&quot;deleting-from-google-photos&quot;&gt;Deleting From Google Photos&lt;/h2&gt;
&lt;p&gt;When you select photos on photos.google.com and press delete, Google asks:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Remove from your Google Account, devices with backup turned on, and places shared within Google Photos?&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;That middle part is the trap: &lt;strong&gt;every phone with Google Photos backup turned on deletes those photos from its gallery too.&lt;/strong&gt; Turn off backup in the Google Photos app (or uninstall it) on every device first.&lt;/p&gt;
&lt;p&gt;There is no &quot;delete everything&quot; button. Select the first photo, scroll far down, Shift+click the last one, delete, repeat. Then &lt;strong&gt;empty the trash&lt;/strong&gt;, or Google keeps everything for 60 more days. Also check &lt;em&gt;Archive&lt;/em&gt; and &lt;em&gt;Locked Folder&lt;/em&gt;, which are not in the main view.&lt;/p&gt;
&lt;p&gt;Keep the unpacked Takeout (or the zips) on a backup disk. Everything you filtered out (screenshots, edits, chat images) exists only there.&lt;/p&gt;
&lt;h2 id=&quot;lessons-learned&quot;&gt;Lessons Learned&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Design the accounts before importing.&lt;/strong&gt; Dedup is per user, and a library&#39;s owner is fixed.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Dry-run every bulk operation on a small sample that overlaps with existing data.&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Google&#39;s Storage saver means byte-level dedup cannot work.&lt;/strong&gt; Match by capture time and name, and confirm with a perceptual hash.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Never trust file times on NFS right after writing.&lt;/strong&gt; Verify through a second mount.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Keep a record of every copy, move and upload&lt;/strong&gt; (one JSON line per file). It made every later step (albums, cleanup, the final deletion) a lookup instead of guesswork.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Move instead of delete, verify, then delete.&lt;/strong&gt; Anything you cannot undo, do last.&lt;/li&gt;
&lt;li&gt;Immich&#39;s job queues are heavy on a small NAS during the initial import: temporarily offloading the microservices to a faster machine works well, but the NAS still briefly fell over when a library scan, 1,500 metadata jobs and a storage migration ran at once. Lower the job concurrency during big imports.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The result: one family archive on the NAS, a personal library per person, phones backing up to Immich, everything reachable from anywhere with no open port at home, and no more Google One subscription.&lt;/p&gt;
</content>
        
    </entry>
    <entry xml:lang="en">
        <title>Fix Severe SSD Write Speed Degradation on Pop!_OS / Linux (LUKS + LVM)</title>
        <published>2026-02-17T00:00:00+00:00</published>
        <updated>2026-02-17T00:00:00+00:00</updated>
        
        <author>
          <name>Unknown</name>
        </author>
        
        <link rel="alternate" type="text/html" href="https://ljufa.iz.rs/posts/ssd-trim-luks-fix/"/>
        <id>https://ljufa.iz.rs/posts/ssd-trim-luks-fix/</id>
        
        <content type="html" xml:base="https://ljufa.iz.rs/posts/ssd-trim-luks-fix/">&lt;h2 id=&quot;the-symptom&quot;&gt;The Symptom&lt;/h2&gt;
&lt;p&gt;Your Linux desktop (especially Pop!_OS or Ubuntu) randomly freezes under heavy write loads. Browsers lock up when downloading files, IDEs (like IntelliJ or VS Code) stutter during indexing, and compiling code brings the system to a halt.&lt;/p&gt;
&lt;h2 id=&quot;before-you-start-benchmark-your-drive&quot;&gt;Before You Start: Benchmark Your Drive&lt;/h2&gt;
&lt;p&gt;Before applying this fix, test your raw SSD write speed. We will write a 5GB file using direct I/O to bypass your RAM cache and measure the actual silicon speed.&lt;/p&gt;
&lt;p&gt;Run this command in your terminal:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color: #CDD6F4; background-color: #1E1E2E;&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: #89B4FA;font-style: italic;&quot;&gt;dd&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt; if=/dev/zero of=test_write_file bs=1G count=&lt;/span&gt;&lt;span style=&quot;color: #FAB387;&quot;&gt;5&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt; oflag=direct&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;em&gt;(Don&#39;t forget to delete the test file afterward: &lt;code&gt;rm test_write_file&lt;/code&gt;)&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The Reality Check:&lt;/strong&gt; If you are using a modern QLC-based SSD (like the Samsung 870 QVO) rated for 500 MB/s, you might see this benchmark return something like &lt;strong&gt;30 MB/s&lt;/strong&gt; (slower than a 15-year-old mechanical hard drive).&lt;/p&gt;
&lt;h2 id=&quot;the-root-cause&quot;&gt;The Root Cause&lt;/h2&gt;
&lt;p&gt;If you enabled &lt;strong&gt;Full Disk Encryption&lt;/strong&gt; during your OS installation, your drive is likely suffering from a blocked TRIM command.&lt;/p&gt;
&lt;p&gt;QLC drives rely on a fast &quot;SLC Cache&quot; (using empty storage space) to achieve high speeds. When you delete a file on Linux, the OS normally sends a &lt;code&gt;TRIM&lt;/code&gt; command to tell the SSD it can reuse those blocks for its fast cache.&lt;/p&gt;
&lt;p&gt;However, &lt;strong&gt;Linux blocks TRIM commands through LUKS encryption by default&lt;/strong&gt; for security reasons. Because the SSD never receives the TRIM command, the hardware thinks the drive is 100% full of undeleted data. The fast cache shrinks to zero, and the drive is forced to write directly to the slow QLC memory, causing your entire OS to freeze while it waits for the disk.&lt;/p&gt;
&lt;h2 id=&quot;how-to-diagnose&quot;&gt;How to Diagnose&lt;/h2&gt;
&lt;p&gt;Run this commands to confirm you are affected:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;1. Check if TRIM is blocked:&lt;/strong&gt;&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color: #CDD6F4; background-color: #1E1E2E;&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: #89B4FA;font-style: italic;&quot;&gt;sudo&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt; fstrim -v /&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;em&gt;If this returns &lt;code&gt;fstrim: /: the discard operation is not supported&lt;/code&gt;, proceed to step 2.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;2. Check if your root drive is encrypted:&lt;/strong&gt;&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color: #CDD6F4; background-color: #1E1E2E;&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: #89B4FA;font-style: italic;&quot;&gt;lsblk&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt; -f&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Look at your root partition (/). If it is nested under a &lt;code&gt;crypto_LUKS&lt;/code&gt; tree, this fix applies to you.&lt;/p&gt;
&lt;hr /&gt;
&lt;h2 id=&quot;the-fix&quot;&gt;The Fix&lt;/h2&gt;
&lt;p&gt;To fix this, we need to explicitly allow the &lt;code&gt;discard&lt;/code&gt; (TRIM) command to pass through the encryption layer.
&lt;em&gt;(Note: This has a minor security tradeoff, as an attacker with physical access to your drive could theoretically determine exactly how much free space you have based on empty blocks. For 99% of desktop users, this is worth the massive performance gain).&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Step 1: Edit your Crypttab&lt;/strong&gt;
Open the cryptography table configuration file:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color: #CDD6F4; background-color: #1E1E2E;&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: #89B4FA;font-style: italic;&quot;&gt;sudo&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt; nano /etc/crypttab&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Find the line corresponding to your encrypted drive (usually named &lt;code&gt;cryptdata&lt;/code&gt;). Add &lt;code&gt;,discard&lt;/code&gt; to the end of the options string.&lt;/p&gt;
&lt;p&gt;It should look something like this:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color: #CDD6F4; background-color: #1E1E2E;&quot; &gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;cryptdata UUID=xxxx-xxxx-xxxx none luks,discard&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;em&gt;(Save and exit: &lt;code&gt;Ctrl+O&lt;/code&gt;, &lt;code&gt;Enter&lt;/code&gt;, &lt;code&gt;Ctrl+X&lt;/code&gt;)&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Step 2: Rebuild the Boot Image&lt;/strong&gt;
Because the drive is unlocked during boot, you must update the initial ramdisk so the system knows about the new rule before the OS loads:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color: #CDD6F4; background-color: #1E1E2E;&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: #89B4FA;font-style: italic;&quot;&gt;sudo&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt; update-initramfs -u -k all&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;strong&gt;Step 3: Reboot&lt;/strong&gt;
You must restart your computer for the encryption layer to reload with the new rules.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Step 4: Dump the Garbage&lt;/strong&gt;
Once you log back in, manually trigger the TRIM command to clear out months/years of dead data:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color: #CDD6F4; background-color: #1E1E2E;&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: #89B4FA;font-style: italic;&quot;&gt;sudo&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt; fstrim -v /&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;em&gt;Note: This might take a few minutes. It should report back that it trimmed many Gigabytes of data! In my case it was ~570Gb!&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Step 5: Enable Automatic Weekly Maintenance&lt;/strong&gt;
Ensure the background systemd timer is enabled so Linux does this for you automatically in the future:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color: #CDD6F4; background-color: #1E1E2E;&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: #89B4FA;font-style: italic;&quot;&gt;sudo&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt; systemctl enable --now fstrim.timer&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;h2 id=&quot;the-result&quot;&gt;The Result&lt;/h2&gt;
&lt;p&gt;Run the &lt;code&gt;dd&lt;/code&gt; benchmark from the beginning of this guide one more time:&lt;/p&gt;
&lt;pre class=&quot;giallo&quot; style=&quot;color: #CDD6F4; background-color: #1E1E2E;&quot; &gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span style=&quot;color: #89B4FA;font-style: italic;&quot;&gt;dd&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt; if=/dev/zero of=test_write_file bs=1G count=&lt;/span&gt;&lt;span style=&quot;color: #FAB387;&quot;&gt;5&lt;/span&gt;&lt;span style=&quot;color: #A6E3A1;&quot;&gt; oflag=direct&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;On a 1TB Samsung 870 QVO, this fix took write speeds from &lt;strong&gt;30.9 MB/s back up to 498 MB/s&lt;/strong&gt; (a 1,600% increase), completely eliminating all system stuttering and lockups.&lt;/p&gt;
</content>
        
    </entry>
</feed>
