Asset is no longer in an import path, removing #13570

norton-chris · 2024-10-18T02:44:37Z

The bug

I created a new postgres and library folder to test it wasn't just something weird there and I scanned in one of the smaller folders to test. I found when I scanned in the library it shows the correct size in the external library tab, but under server stats it says 0B but have 1400 images and 200 videos. But I can see everything in Immich just fine. But if I try to rescan the folder, I get a bunch of messages that says:
"DEBUG [Microservices:LibraryService] Asset is no longer in an import path, removing: " then the file name it's removing. Then I go back to immich Photos tab and all the images are gone, but they still exist on the hard drive and are in the correct spot in the immich_server docker container.

I have the most recent version of Immich installed through Docker on Armbian on my RockPi. It has run without issues till a couple weeks ago.

My question is why is it removing them? and it was scanning them correctly and the path is the same, but what changed?

How do I fix this issue? I'm out of ideas.

The OS that Immich Server is running on

Armbian 23.8.1 bullseye

Version of Immich Server

v1.118.2

Version of Immich Mobile App

v1.118.0

Platform with the issue

Server
Web
Mobile

Your docker-compose.yml content

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    extends:
      file: hwaccel.transcoding.yml
      service: rkmpp # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /srv/dev-disk-by-uuid-7936f8bd-919b-463a-8e0f-ea540e30d588/RAID_Drive/Family_Photos:/usr/src/app/external
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - '2283:2283'
    depends_on:
      - redis
      - database
    restart: always
    healthcheck:
      disable: false

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}-armnn
    extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
      file: hwaccel.ml.yml
      service: armnn # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always
    healthcheck:
      disable: false

  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:2ba50e1ac3a0ea17b736ce9db2b0a9f6f8b85d4c27d5f5accc6a416d8f42c6d5
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    healthcheck:
      test: pg_isready --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1
      interval: 5m
      start_interval: 30s
      start_period: 5m
    command:
      [
        'postgres',
        '-c',
        'shared_preload_libraries=vectors.so',
        '-c',
        'search_path="$$user", public, vectors',
        '-c',
        'logging_collector=on',
        '-c',
        'max_wal_size=2GB',
        '-c',
        'shared_buffers=512MB',
        '-c',
        'wal_compression=on',
      ]
    restart: always

volumes:
  model-cache:

Your .env content

# The location where your uploaded files are stored
UPLOAD_LOCATION=/srv/dev-disk-by-uuid-7936f8bd-919b-463a-8e0f-ea540e30d588/RAID_Drive/library
# The location where your database files are stored
DB_DATA_LOCATION=/srv/dev-disk-by-uuid-7936f8bd-919b-463a-8e0f-ea540e30d588/RAID_Drive/postgres

# To set a timezone, uncomment the next line and change Etc/UTC to a TZ identifier from this list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List
TZ=America/Chicago

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
DB_PASSWORD={password}

# The values below this line do not need to be changed
###################################################################################
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich
REDIS_HOSTNAME=immich_redis

Reproduction steps

Start fresh Immich, move progres and library to backup folder, then create new postgres and library folder. Start and setup Immich
Scan in external library
Check to make sure all the photos and videos are on Immich
Rescan and watch logs.
Photos and videos will start disappearing from Immich and lots of "Asset is no longer in an import path, removing" messages.

Relevant log output

[Nest] 7  - 10/17/2024, 8:14:09 PM     LOG [Microservices:LibraryService] Refreshing library c575317b-eaab-42fa-b5ad-eddcff680814 for new assets
[Nest] 7  - 10/17/2024, 8:14:09 PM     LOG [Microservices:LibraryService] Scanning library c575317b-eaab-42fa-b5ad-eddcff680814 for removed assets
[Nest] 7  - 10/17/2024, 8:14:10 PM   DEBUG [Microservices:LibraryService] Discovered 6062 asset(s) on disk for library c575317b-eaab-42fa-b5ad-eddcff680814...
[Nest] 17  - 10/17/2024, 8:14:11 PM   DEBUG [Api:LoggingInterceptor~2046rh7w] GET /api/jobs 200 26.40ms 192.168.1.1
[Nest] 17  - 10/17/2024, 8:14:11 PM   DEBUG [Api:LoggingInterceptor~2vkwe71v] GET /api/system-config 200 30.51ms 192.168.1.1
[Nest] 7  - 10/17/2024, 8:14:11 PM   DEBUG [Microservices:LibraryService] Discovered 6062 asset(s) in library c575317b-eaab-42fa-b5ad-eddcff680814...
[Nest] 17  - 10/17/2024, 8:14:12 PM   DEBUG [Api:LoggingInterceptor~g502mb7d] GET /api/jobs 200 38.02ms 192.168.1.1
[Nest] 7  - 10/17/2024, 8:14:12 PM   DEBUG [Microservices:LibraryService] Finished queueing scan of 6062 assets on disk for library c575317b-eaab-42fa-b5ad-eddcff680814
[Nest] 17  - 10/17/2024, 8:14:12 PM   DEBUG [Api:LoggingInterceptor~ws85pfep] GET /api/server/storage 200 1.89ms 192.168.1.1
[Nest] 17  - 10/17/2024, 8:14:12 PM   DEBUG [Api:LoggingInterceptor~y86nh6iw] GET /api/server/about 200 88.90ms 192.168.1.1
[Nest] 17  - 10/17/2024, 8:14:12 PM   DEBUG [Api:LoggingInterceptor~htenypsl] GET /api/server/storage 200 0.90ms 192.168.1.1
[Nest] 17  - 10/17/2024, 8:14:12 PM   DEBUG [Api:LoggingInterceptor~50lsv7op] GET /api/server/version-history 200 30.91ms 192.168.1.1
[Nest] 17  - 10/17/2024, 8:14:12 PM   DEBUG [Api:LoggingInterceptor~0lvv5o71] GET /api/server/about 200 92.30ms 192.168.1.1
[Nest] 7  - 10/17/2024, 8:14:12 PM   DEBUG [Microservices:LibraryService] Queued check of 6062 asset(s) in library c575317b-eaab-42fa-b5ad-eddcff680814...
[Nest] 7  - 10/17/2024, 8:14:12 PM     LOG [Microservices:LibraryService] Finished queueing check of 6062 assets for library c575317b-eaab-42fa-b5ad-eddcff680814
[Nest] 17  - 10/17/2024, 8:14:12 PM   DEBUG [Api:LoggingInterceptor~9o03xz5n] GET /api/jobs 200 207.28ms 192.168.1.1
[Nest] 17  - 10/17/2024, 8:14:17 PM   DEBUG [Api:LoggingInterceptor~73756l1t] GET /api/jobs 200 26.58ms 192.168.1.1
[Nest] 17  - 10/17/2024, 8:14:22 PM   DEBUG [Api:LoggingInterceptor~21gls737] GET /api/jobs 200 53.13ms 192.168.1.1
[Nest] 7  - 10/17/2024, 8:14:23 PM   DEBUG [Microservices:LibraryService] Asset is no longer in an import path, removing: /usr/src/app/external/Norton_Family/HOW_TO_DELETE_GOOGLE_PHOTOS.mov
[Nest] 7  - 10/17/2024, 8:14:23 PM   DEBUG [Microservices:LibraryService] Asset is no longer in an import path, removing: /usr/src/app/external/Norton_Family/Boundary Waters 2011/IMG_1027.JPG
[Nest] 7  - 10/17/2024, 8:14:23 PM   DEBUG [Microservices:LibraryService] Asset is no longer in an import path, removing: /usr/src/app/external/Norton_Family/Boundary Waters 2011/IMG_1025.JPG

Additional information

Immich has been working fine since January and I've been staying up-to-date. I've followed the breaking changes and made appropriate changes.

alextran1502 · 2024-10-18T08:02:23Z

Can you help updating to the latest version first then see of the error message is still there?

norton-chris · 2024-10-18T15:26:32Z

Ope thought I was at the latest, you guys come out with releases quick :) I just upgraded to v1.118.2 and did the appropriate yml config changes, then deleted the library, created and scanned new library, and I saw all the photos and videos and 154GB in the external library tab, however server stats only says 2GB. Then did a rescan and got the same issue as before. Tons of messages saying: "DEBUG [Microservices:LibraryService] Asset is no longer in an import path, removing: " then the photo name.
The external library says 0 assets 0B after that. I think the 2GB in the server stats is from my phone backing up to the library/upload folder, I don't think it's seeing any of the external library assets.
edit: I updated the version and docker-compose.yml in the issue.

etnoy · 2024-10-18T16:39:15Z

What is your import path?

norton-chris · 2024-10-18T17:06:43Z

@etnoy
on Immich:
"external/Norton_Family"
docker compose:
/srv/dev-disk-by-uuid-7936f8bd-919b-463a-8e0f-ea540e30d588/RAID_Drive/Family_Photos:/usr/src/app/external

docker container shows it exists:
docker exec -it immich_server bash
root@de580e9e8905:/usr/src/app# cd external/
root@de580e9e8905:/usr/src/app/external# ls
Norton_Family
(other folders I excluded from this output)

etnoy · 2024-10-18T19:29:02Z

Your import path must be absolute.

/usr/src/app/external/Norton_Family

I didn't know a relative path would pass validation, that part is a bug

norton-chris · 2024-10-18T22:52:29Z

Oh. Well, that was an easy fix. I see these messages during a rescan and they are showing in the external library tab:
"DEBUG [Microservices:LibraryService] Asset was offline or modified, updating asset record"

So it looks like it's working alright now after using an absolute path. Thanks!

Weird that's been working fine for me for months, didn't realize it was absolute path, but I should've tried that.

etnoy · 2024-10-18T23:03:44Z

Oh. Well, that was an easy fix. I see these messages during a rescan and they are showing in the external library tab: "DEBUG [Microservices:LibraryService] Asset was offline or modified, updating asset record"

So it looks like it's working alright now after using an absolute path. Thanks!

Weird that's been working fine for me for months, didn't realize it was absolute path, but I should've tried that.

We have done major work under the hood recently, likely it worked before but no longer. Closing.

etnoy closed this as completed Oct 18, 2024

etnoy mentioned this issue Oct 21, 2024

fix(server): only allow absolute import paths #13642

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Asset is no longer in an import path, removing #13570

Asset is no longer in an import path, removing #13570

norton-chris commented Oct 18, 2024 •

edited

Loading

alextran1502 commented Oct 18, 2024

norton-chris commented Oct 18, 2024 •

edited

Loading

etnoy commented Oct 18, 2024

norton-chris commented Oct 18, 2024

etnoy commented Oct 18, 2024

norton-chris commented Oct 18, 2024

etnoy commented Oct 18, 2024

Asset is no longer in an import path, removing #13570

Asset is no longer in an import path, removing #13570

Comments

norton-chris commented Oct 18, 2024 • edited Loading

The bug

The OS that Immich Server is running on

Version of Immich Server

Version of Immich Mobile App

Platform with the issue

Your docker-compose.yml content

Your .env content

Reproduction steps

Relevant log output

Additional information

alextran1502 commented Oct 18, 2024

norton-chris commented Oct 18, 2024 • edited Loading

etnoy commented Oct 18, 2024

norton-chris commented Oct 18, 2024

etnoy commented Oct 18, 2024

norton-chris commented Oct 18, 2024

etnoy commented Oct 18, 2024

norton-chris commented Oct 18, 2024 •

edited

Loading

norton-chris commented Oct 18, 2024 •

edited

Loading