feat (web/server) 360 degrees Web panoramas #3098

dmitry-brazhenko · 2023-07-03T10:41:14Z

In this PR I aded

Support for 360 Panoramas as it was discussed here: [Feature]: Panoramas: Use PhotoSphere Viewer to display photos with equirectangular projection #1630
Minor technical changes and improvements

How it was implemented

Once an images is uploaded, in metadata-extraction.processor.ts it checks aspectRatio. If it is aspectRatio >= 2, we assume that it is a panorama. This can be potentialy improved, but looks like a common way to determine if image is a panorama or not
If it is a panorama, we add a prtoperty isPanorama: true
We use library @egjs/svelte-view360 to show the panoram
In the library we use a special symbol to display if a thumb is a panorama.

Minor technical changes and improvements:

restart:always for all docker containers in development docker-compose
Fixed links in database-migrations.md file

Potential further improvements:

Probably we can switch library and use smth else. But this liibrary works OK both on desktop and mobile-web
Support panoramas in Flutter mobile app
There 360 videos. Later we can support them as well

Here is thumb that shows a panorama:

Panorama after is opened:

vercel · 2023-07-03T10:41:19Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
immich	⬜️ Ignored (Inspect)	Visit Preview		Jul 23, 2023 5:54pm

dmitry-brazhenko · 2023-07-03T15:51:21Z

Probably I need to add some tests, but not sure. Need some advise.
Also I am not 100% sure that I properly fixed open-api-doc. Please help me with this stuff.
Thanks

jrasm91

You can try running make api to regenerate the api. You can also run the npm run check:all scripts in the web/server folders to make sure those all pass.

web/src/lib/components/assets/thumbnail/360arrow.svg

mobile/openapi/pubspec.yaml

dmitry-brazhenko · 2023-07-03T17:36:12Z

For further steps for improvement we can add a "panorama" filter here:

web/src/lib/components/assets/thumbnail/image-thumbnail.svelte

dmitry-brazhenko · 2023-07-04T17:41:55Z

Hi!
Do you have any ideas what can be improved in this PR? Can we merge it? :)

alextran1502 · 2023-07-04T17:55:08Z

@dmitry-brazhenko hello, I haven't gotten a chance to review the PR yet. Once I do and if there is nothing else to address, it will be merged. Thank you for the contribution

server/src/microservices/processors/metadata-extraction.processor.ts

jrasm91 · 2023-07-04T18:00:49Z

web/src/lib/components/asset-viewer/panorama-viewer.css

Is it possible to include this css in the asset-viewer.svelte component directly via a <style> ... </style> tag at the end of the file instead?

I tried to do that, but it did not work. Here is a lib example: https://github.com/naver/egjs-view360/blob/master/packages/svelte-view360/demo/App.svelte

I overrided original css file. I asked a question here, but the suggestion did not work.
I am not very good in frontend technologies. If there are any ways to enable, please let me know, I will fix it :)

I think it can work, but you might need to add something special to make the style not scoped to the component.

web/src/lib/components/asset-viewer/panorama-viewer.svelte

web/src/lib/components/assets/thumbnail/image-thumbnail.svelte

…sor.ts Co-authored-by: Jason Rasmussen <[email protected]>

jrasm91

Need to double check the API generated code, but other than that LGTM.

jrasm91 · 2023-07-04T22:52:49Z

server/openapi-generator/templates/web/apiInner.mustache

-                localVarFormParams.append('{{baseName}}', {{paramName}} as any);{{/isPrimitiveType}}{{^isPrimitiveType}}{{#isEnum}}
-                localVarFormParams.append('{{baseName}}', {{paramName}} as any);{{/isEnum}}{{^isEnum}}
-                localVarFormParams.append('{{baseName}}', new Blob([JSON.stringify({{paramName}})], { type: "application/json", }));{{/isEnum}}{{/isPrimitiveType}}{{/multipartFormData}}
+                localVarFormParams.append('{{baseName}}', {{paramName}} as any);{{/isPrimitiveType}}{{^isPrimitiveType}}
+                localVarFormParams.append('{{baseName}}', new Blob([JSON.stringify({{paramName}})], { type: "application/json", }));{{/isPrimitiveType}}{{/multipartFormData}}


Need to double check where this change is coming from

alextran1502 · 2023-07-08T20:32:41Z

I think there is some issue with the recent rebase/merge, can you only include the changes to the panorama view?

dmitry-brazhenko · 2023-07-08T20:32:41Z

After I ran "make api" - it resulted to multiple file changes

alextran1502 · 2023-07-08T20:36:59Z

These changes somehow reverting some of the changes that we made. How about you only include the changes in the server and mobile and leave the API generation for me?

dmitry-brazhenko · 2023-07-08T20:43:17Z

Yes, I will do that

dmitry-brazhenko · 2023-07-08T20:48:37Z

@alextran1502 I reverted my attempts to merge main branch to my branch.
Could you help me with merging please?
Thanks

alextran1502 · 2023-07-09T02:25:27Z

server/openapi-generator/templates/web/apiInner.mustache

Please remove this file from your PR

alextran1502 · 2023-07-09T02:25:36Z

docker/docker-compose.dev.yml

Please remove this file from your PR

Why not to keep it?
Once I started running Immich, it was failing as containers just stopped. Adding this restart;always helped me to get started.

btw, I removed it for now, but I can create a separate PR and merge it there

alextran1502 · 2023-07-09T02:25:46Z

docs/docs/developer/database-migrations.md

Please remove this file from your PR

Why remove it? I fixed the path?
https://github.com/immich-app/immich/tree/main/server/src/infra/entities

btw, I reverted it, but I can create a separate PR and merge it there

dmitry-brazhenko · 2023-07-09T18:54:21Z

@alextran1502 would you mind if merge docker compose improvement and MD fix in a separate PR?

uhthomas · 2023-07-09T20:41:17Z

Fixes #2610 and #1630.

brighteyed · 2023-07-16T13:05:43Z

If it is aspectRatio >= 2, we assume that it is a panorama

I think 360 panoramas should be detected more reliably in this PR. Currently, the code assumes that all images with an aspect ratio >= 2 are 360 panoramas.

But, for example:

Screenshots with landscape orientation should not be detected as 360 panoramas
Non-360 panoramas (for example, with cylindrical projection) should not be detected as 360 panoramas

I believe we should use the EXIF tag ProjectionType to detect 360 panoramas in addition to the aspect ratio check

CodeWithMa · 2023-07-16T16:55:24Z

If it is aspectRatio >= 2, we assume that it is a panorama

I think 360 panoramas should be detected more reliably in this PR. Currently, the code assumes that all images with an aspect ratio >= 2 are 360 panoramas.

But, for example:
* Screenshots with landscape orientation should not be detected as 360 panoramas

* Non-360 panoramas (for example, with cylindrical projection) should not be detected as 360 panoramas
I believe we should use the EXIF tag ProjectionType to detect 360 panoramas in addition to the aspect ratio check

I agree. My google pixel photosphere images have the following projection type:

Projection Type                 : equirectangular

I also have photoshere images that are not complete 360 degree images so the aspect ratio is not 2:1.

dmitry-brazhenko · 2023-07-23T17:55:54Z

@CodeWithMa @brighteyed you're right. I fixed panorama detection. Now it consumes Exif and marks images by "panorama" only if there is a necessary exif tag.

@alextran1502 Could you help me with merging this PR?

Thanks

alextran1502 · 2023-07-23T18:14:43Z

Can you create another PR to consolidate the changes on the web and the server? Please put migrations into a single file. Once you've done that please ping me and I can help generating the API

brighteyed

If we transform isPanorama fields into projectionType it will be easier in the future to add additional types of projections (most likely cylindrical)

brighteyed · 2023-07-23T19:03:30Z

server/src/domain/asset/response-dto/asset-response.dto.ts

@@ -32,6 +32,7 @@ export class AssetResponseDto {
  people?: PersonResponseDto[];
  /**base64 encoded sha1 hash */
  checksum!: string;
+  isPanorama!: boolean;


I'd prefer projectionType field with the enum type

brighteyed · 2023-07-23T19:06:31Z

server/src/domain/asset/response-dto/asset-response.dto.ts

@@ -58,6 +59,7 @@ export function mapAsset(entity: AssetEntity): AssetResponseDto {
    tags: entity.tags?.map(mapTag),
    people: entity.faces?.map(mapFace),
    checksum: entity.checksum.toString('base64'),
+    isPanorama: entity.isPanorama,


I'd prefer projectionType field with the enum type

brighteyed · 2023-07-23T19:09:08Z

server/src/infra/entities/exif.entity.ts

@@ -43,6 +43,9 @@ export class ExifEntity {
  @Column({ type: 'float', nullable: true })
  longitude!: number | null;

+  @Column({ type: 'varchar', nullable: true })
+  projectionType!: string | null;


Can we use enum here?

server/package.json

dmitry-brazhenko · 2023-07-24T09:52:07Z

Will go on in this PR: #3412.

Implemented Projectiontype as enum

dmitry-brazhenko added 3 commits July 3, 2023 11:19

panorama viewer.svelte

7ca9470

restart for all services in dev docker compose

e69b633

Propagated propery isPanorama

5d31716

dmitry-brazhenko added 3 commits July 3, 2023 13:54

db migration + changed doc

e12e000

working on uploading panoramas

4690160

nice 360 arrow if image is 360 panorama

1b56a38

dmitry-brazhenko marked this pull request as ready for review July 3, 2023 15:51

dmitry-brazhenko changed the title ~~[Draft] 360 degrees Web panoramas~~ 360 degrees Web panoramas Jul 3, 2023

dmitry-brazhenko added 2 commits July 3, 2023 18:15

minor fixes

e13a3cc

OpenAPI Generator result

73b6d8a

jrasm91 reviewed Jul 3, 2023

View reviewed changes

web/src/lib/components/assets/thumbnail/360arrow.svg Outdated Show resolved Hide resolved

alextran1502 reviewed Jul 3, 2023

View reviewed changes

mobile/openapi/pubspec.yaml Show resolved Hide resolved

api regenerated + standard collection 360svg

f1bd1d6

jrasm91 reviewed Jul 3, 2023

View reviewed changes

web/src/lib/components/assets/thumbnail/image-thumbnail.svelte Outdated Show resolved Hide resolved

<Rotate360Icon size="24"/>

f638fce

jrasm91 reviewed Jul 4, 2023

View reviewed changes

dmitry-brazhenko and others added 5 commits July 5, 2023 00:12

assetData -> dataUrl

ebacdd2

Moved panorama icon to "thumbnail"

4398ec0

fix formatting

dea4f38

Merge branch 'main' into web_panoramas

a100f40

Update server/src/microservices/processors/metadata-extraction.proces…

c6d23e8

…sor.ts Co-authored-by: Jason Rasmussen <[email protected]>

jrasm91 approved these changes Jul 4, 2023

View reviewed changes

dmitry-brazhenko changed the title ~~360 degrees Web panoramas~~ feat (web/server) 360 degrees Web panoramas Jul 6, 2023

vercel bot temporarily deployed to Preview July 8, 2023 20:21 Inactive

dmitry-brazhenko force-pushed the web_panoramas branch 2 times, most recently from 2654895 to c6d23e8 Compare July 8, 2023 20:47

alextran1502 reviewed Jul 9, 2023

View reviewed changes

server/openapi-generator/templates/web/apiInner.mustache

Copy link

Contributor

alextran1502 Jul 9, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove this file from your PR

alextran1502 reviewed Jul 9, 2023

View reviewed changes

dmitry-brazhenko added 3 commits July 9, 2023 07:58

removed "always" in github dev

2fd30d8

reverted apiinned.mustache

fb5e1b9

reverted database-migrations.md

f56273a

vercel bot temporarily deployed to Preview July 9, 2023 06:05 Inactive

fix for panorama detection

1f8e156

brighteyed reviewed Jul 23, 2023

View reviewed changes

dmitry-brazhenko closed this Jul 24, 2023

dmitry-brazhenko mentioned this pull request Jul 24, 2023

feat (web/server) 360 degrees Web panoramas [attempt 2] #3412

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat (web/server) 360 degrees Web panoramas #3098

feat (web/server) 360 degrees Web panoramas #3098

dmitry-brazhenko commented Jul 3, 2023 •

edited

Loading

vercel bot commented Jul 3, 2023 •

edited

Loading

dmitry-brazhenko commented Jul 3, 2023

jrasm91 left a comment

dmitry-brazhenko commented Jul 3, 2023 •

edited

Loading

dmitry-brazhenko commented Jul 4, 2023

alextran1502 commented Jul 4, 2023

jrasm91 Jul 4, 2023

dmitry-brazhenko Jul 4, 2023

jrasm91 Jul 4, 2023

jrasm91 left a comment

jrasm91 Jul 4, 2023

alextran1502 commented Jul 8, 2023

dmitry-brazhenko commented Jul 8, 2023

alextran1502 commented Jul 8, 2023

dmitry-brazhenko commented Jul 8, 2023

dmitry-brazhenko commented Jul 8, 2023

alextran1502 Jul 9, 2023

alextran1502 Jul 9, 2023

dmitry-brazhenko Jul 9, 2023 •

edited

Loading

alextran1502 Jul 9, 2023

dmitry-brazhenko Jul 9, 2023 •

edited

Loading

dmitry-brazhenko commented Jul 9, 2023 •

edited

Loading

uhthomas commented Jul 9, 2023 •

edited

Loading

brighteyed commented Jul 16, 2023

CodeWithMa commented Jul 16, 2023

dmitry-brazhenko commented Jul 23, 2023 •

edited

Loading

alextran1502 commented Jul 23, 2023

brighteyed left a comment

brighteyed Jul 23, 2023

brighteyed Jul 23, 2023

brighteyed Jul 23, 2023

dmitry-brazhenko commented Jul 24, 2023

feat (web/server) 360 degrees Web panoramas #3098

feat (web/server) 360 degrees Web panoramas #3098

Conversation

dmitry-brazhenko commented Jul 3, 2023 • edited Loading

vercel bot commented Jul 3, 2023 • edited Loading

dmitry-brazhenko commented Jul 3, 2023

jrasm91 left a comment

Choose a reason for hiding this comment

dmitry-brazhenko commented Jul 3, 2023 • edited Loading

dmitry-brazhenko commented Jul 4, 2023

alextran1502 commented Jul 4, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jrasm91 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alextran1502 commented Jul 8, 2023

dmitry-brazhenko commented Jul 8, 2023

alextran1502 commented Jul 8, 2023

dmitry-brazhenko commented Jul 8, 2023

dmitry-brazhenko commented Jul 8, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmitry-brazhenko Jul 9, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmitry-brazhenko Jul 9, 2023 • edited Loading

Choose a reason for hiding this comment

dmitry-brazhenko commented Jul 9, 2023 • edited Loading

uhthomas commented Jul 9, 2023 • edited Loading

brighteyed commented Jul 16, 2023

CodeWithMa commented Jul 16, 2023

dmitry-brazhenko commented Jul 23, 2023 • edited Loading

alextran1502 commented Jul 23, 2023

brighteyed left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmitry-brazhenko commented Jul 24, 2023

dmitry-brazhenko commented Jul 3, 2023 •

edited

Loading

vercel bot commented Jul 3, 2023 •

edited

Loading

dmitry-brazhenko commented Jul 3, 2023 •

edited

Loading

dmitry-brazhenko Jul 9, 2023 •

edited

Loading

dmitry-brazhenko Jul 9, 2023 •

edited

Loading

dmitry-brazhenko commented Jul 9, 2023 •

edited

Loading

uhthomas commented Jul 9, 2023 •

edited

Loading

dmitry-brazhenko commented Jul 23, 2023 •

edited

Loading