compreface-core error: E tensorflow/stream_executor/cuda/cuda_driver.cc:351] failed call to cuInit: UNKNOWN ERROR (303)

[sirhideo]

Hi, for some reason my compreface docker doesn't work anymore. I am using double-take to get compreface process the images. Whenever an images needs to be processed compreface-core shows the captioned error message and never process anything.

I have tried to docker system prune -a and re-download compreface docker images again, and also remove the postgresql database volume. Any idea what's going on please?

[pospielov]

Could you provide:

1. Your GPU name
2. your CompreFace build and version

[sirhideo]

Hello,

1. I am using CPU at the moment
2. I have tried both
   #CORE_VERSION=1.1.0-arcface-r100
   CORE_VERSION=1.1.0-facenet
   Right now mainly using facenet.

[pospielov]

Could you run docker compose logs -f and send the result here?

[sirhideo]

@pospielov sorry for late reply. here is the logs after startup docker

Attaching to compreface-ui, compreface-admin, compreface-api, compreface-core, compreface-postgres-db compreface-admin | Listening for transport dt_socket at address: 5005 compreface-admin | compreface-admin | . ____ _ __ _ _ compreface-admin | /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \ compreface-admin | ( ( )\___ | '_ | '_| | '_ \/ _ | \ \ \
compreface-admin | \/ )| |)| | | | | || (| | ) ) ) )
compreface-admin | ' || .__|| ||| |_, | / / / /
compreface-admin | =========||==============|/=////
compreface-admin | :: Spring Boot :: (v2.5.13)
compreface-admin |
compreface-admin | 2023-07-15 15:10:53.925 INFO 7 --- [kground-preinit] o.h.validator.internal.util.Version : HV000001: Hibernate Validator 6.2.3.Final
compreface-admin | 2023-07-15 15:10:53.961 INFO 7 --- [ main] com.exadel.frs.FrsApplication : Starting FrsApplication v0.0.1-SNAPSHOT using Java 11.0.8 on fac46ff53d70 with PID 7 (/home/app.jar started by root in /)
compreface-admin | 2023-07-15 15:10:53.962 DEBUG 7 --- [ main] com.exadel.frs.FrsApplication : Running with Spring Boot v2.5.13, Spring v5.3.19
compreface-admin | 2023-07-15 15:10:53.962 INFO 7 --- [ main] com.exadel.frs.FrsApplication : The following 1 profile is active: "dev"
compreface-api | Listening for transport dt_socket at address: 5005
compreface-api |
compreface-api | . ____ _ __ _ _
compreface-api | /\ / ' __ _ () __ __ _ \ \ \
compreface-api | ( ( )__ | ' | '| | ' / _ | \ \ \ \ compreface-api | \\/ ___)| |_)| | | | | || (_| | ) ) ) ) compreface-api | ' |____| .__|_| |_|_| |_\__, | / / / / compreface-api | =========|_|==============|___/=/_/_/_/ compreface-api | :: Spring Boot :: (v2.5.13) compreface-api | compreface-api | 2023-07-15 15:10:53.856 INFO 7 --- [kground-preinit] o.h.validator.internal.util.Version : HV000001: Hibernate Validator 6.2.3.Final compreface-api | 2023-07-15 15:10:53.905 INFO 7 --- [ main] com.exadel.frs.TrainServiceApplication : Starting TrainServiceApplication v0.0.1-SNAPSHOT using Java 11.0.8 on dc208165642e with PID 7 (/home/app.jar started by root in /) compreface-api | 2023-07-15 15:10:53.907 INFO 7 --- [ main] com.exadel.frs.TrainServiceApplication : The following 1 profile is active: "dev" compreface-api | 2023-07-15 15:10:54.082 WARN 7 --- [ main] o.s.b.c.config.ConfigDataEnvironment : Property 'spring.profiles' imported from location 'class path resource [application.yml]' is invalid and should be replaced with 'spring.config.activate.on-profile' [origin: class path resource [application.yml] from app.jar - 94:13] compreface-postgres-db | 2023-07-15 15:10:50.252 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432 compreface-postgres-db | 2023-07-15 15:10:50.252 UTC [1] LOG: listening on IPv6 address "::", port 5432 compreface-postgres-db | 2023-07-15 15:10:50.260 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" compreface-postgres-db | 2023-07-15 15:10:50.282 UTC [25] LOG: database system was shut down at 2023-07-15 15:10:44 UTC compreface-postgres-db | 2023-07-15 15:10:50.289 UTC [1] LOG: database system is ready to accept connections compreface-core | [uWSGI] getting INI configuration from uwsgi.ini compreface-core | *** Starting uWSGI 2.0.19 (64bit) on [Sat Jul 15 15:10:50 2023] *** compreface-core | compiled with version: 10.2.1 20210110 on 07 October 2022 10:36:44 compreface-core | os: Linux-5.19.0-46-generic #47~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Jun 21 15:35:31 UTC 2 compreface-core | nodename: 5f6d0b944310 compreface-core | machine: x86_64 compreface-core | clock source: unix compreface-core | detected number of CPU cores: 8 compreface-core | current working directory: /app/ml compreface-core | detected binary path: /usr/local/bin/uwsgi compreface-core | !!! no internal routing support, rebuild with pcre support !!! compreface-core | setgid() to 33 compreface-core | setuid() to 33 compreface-core | your memory page size is 4096 bytes compreface-core | detected max file descriptor number: 1048576 compreface-core | lock engine: pthread robust mutexes compreface-core | thunder lock: disabled (you can enable it with --thunder-lock) compreface-core | uwsgi socket 0 bound to TCP address 0.0.0.0:3000 fd 3 compreface-core | Python version: 3.7.14 (default, Oct 5 2022, 16:16:25) [GCC 10.2.1 20210110] compreface-core | Python main interpreter initialized at 0x55c8daa7bc60 compreface-core | python threads support enabled compreface-core | your server socket listen backlog is limited to 100 connections compreface-core | your mercy for graceful operations on workers is 60 seconds compreface-core | mapped 218712 bytes (213 KB) for 2 cores compreface-core | *** Operational MODE: preforking *** compreface-core | {"severity": "INFO", "message": "{'BUILD_VERSION': 'dev', 'CALCULATION_PLUGIN': 'facenet.Calculator', 'EXTRA_PLUGINS': ['facenet.LandmarksDetector', 'agegender.GenderDetector', 'agegender.AgeDetector', 'facenet.facemask.MaskDetector', 'facenet.PoseEstimator'], 'FACE_DETECTION_PLUGIN': 'facenet.FaceDetector', 'GPU_IDX': -1, 'IMG_LENGTH_LIMIT': 640, 'INTEL_OPTIMIZATION': False, 'IS_DEV_ENV': False, 'LOGGING_LEVEL_NAME': 'DEBUG', 'ML_PORT': 3000, 'RUN_MODE': True}", "request": null, "logger": "src.app", "module": "app", "traceback": null, "build_version": "dev"} compreface-core | {"severity": "DEBUG", "message": "Creating new app for WSGI", "request": null, "logger": "src.app", "module": "app", "traceback": null, "build_version": "dev"} compreface-core | WSGI app 0 (mountpoint='') ready in 3 seconds on interpreter 0x55c8daa7bc60 pid: 1 (default app) compreface-core | *** uWSGI is running in multiple interpreter mode *** compreface-core | spawned uWSGI master process (pid: 1) compreface-core | spawned uWSGI worker 1 (pid: 23, cores: 1) compreface-core | spawned uWSGI worker 2 (pid: 24, cores: 1) compreface-ui | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration compreface-ui | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/ compreface-ui | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh compreface-ui | 10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf is not a file or does not exist compreface-ui | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh compreface-ui | 20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/nginx.conf.template to /etc/nginx/conf.d/nginx.conf compreface-ui | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh compreface-ui | /docker-entrypoint.sh: Configuration complete; ready for start up compreface-admin | 2023-07-15 15:10:57.507 INFO 7 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode. compreface-admin | 2023-07-15 15:10:57.720 INFO 7 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 195 ms. Found 11 JPA repository interfaces. compreface-api | 2023-07-15 15:10:58.101 INFO 7 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode. compreface-admin | 2023-07-15 15:10:58.210 INFO 7 --- [ main] o.s.cloud.context.scope.GenericScope : BeanFactory id=87eecb33-b947-31b8-bbb7-8d063df92185 compreface-api | 2023-07-15 15:10:58.409 INFO 7 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 278 ms. Found 8 JPA repository interfaces. compreface-admin | 2023-07-15 15:10:59.498 INFO 7 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http) compreface-admin | 2023-07-15 15:10:59.520 INFO 7 --- [ main] o.a.coyote.http11.Http11NioProtocol : Initializing ProtocolHandler ["http-nio-8080"] compreface-admin | 2023-07-15 15:10:59.521 INFO 7 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat] compreface-admin | 2023-07-15 15:10:59.522 INFO 7 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.62] compreface-admin | 2023-07-15 15:10:59.676 INFO 7 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext compreface-admin | 2023-07-15 15:10:59.676 INFO 7 --- [ main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 5584 ms compreface-api | 2023-07-15 15:11:00.339 INFO 7 --- [ main] o.s.cloud.context.scope.GenericScope : BeanFactory id=9776d44d-52e1-38dc-b671-bd86977f22bf compreface-admin | 2023-07-15 15:11:00.648 INFO 7 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting... compreface-admin | 2023-07-15 15:11:00.829 INFO 7 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Start completed. compreface-admin | 2023-07-15 15:11:01.073 INFO 7 --- [ main] liquibase.database : Set default schema name to public compreface-admin | 2023-07-15 15:11:01.203 INFO 7 --- [ main] liquibase.lockservice : Successfully acquired change log lock compreface-api | 2023-07-15 15:11:01.463 INFO 7 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'cacheConfig' of type [com.exadel.frs.core.trainservice.config.CacheConfig$$EnhancerBySpringCGLIB$$b9639fd0] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) compreface-admin | 2023-07-15 15:11:02.388 INFO 7 --- [ main] liquibase.changelog : Reading from public.databasechangelog compreface-api | 2023-07-15 15:11:02.651 INFO 7 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http) compreface-api | 2023-07-15 15:11:02.679 INFO 7 --- [ main] o.a.coyote.http11.Http11NioProtocol : Initializing ProtocolHandler ["http-nio-8080"] compreface-api | 2023-07-15 15:11:02.681 INFO 7 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat] compreface-api | 2023-07-15 15:11:02.681 INFO 7 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.62] compreface-admin | 2023-07-15 15:11:02.816 INFO 7 --- [ main] liquibase.lockservice : Successfully released change log lock compreface-api | 2023-07-15 15:11:02.951 INFO 7 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext compreface-api | 2023-07-15 15:11:02.951 INFO 7 --- [ main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 8867 ms compreface-postgres-db | 2023-07-15 15:11:03.214 UTC [32] WARNING: there is no transaction in progress compreface-admin | 2023-07-15 15:11:03.428 INFO 7 --- [ main] o.hibernate.jpa.internal.util.LogHelper : HHH000204: Processing PersistenceUnitInfo [name: default] compreface-admin | 2023-07-15 15:11:03.599 INFO 7 --- [ main] org.hibernate.Version : HHH000412: Hibernate ORM core version 5.4.33 compreface-admin | 2023-07-15 15:11:03.917 INFO 7 --- [ main] o.hibernate.annotations.common.Version : HCANN000001: Hibernate Commons Annotations {5.1.2.Final} compreface-api | 2023-07-15 15:11:04.072 INFO 7 --- [ main] o.hibernate.jpa.internal.util.LogHelper : HHH000204: Processing PersistenceUnitInfo [name: default] compreface-admin | 2023-07-15 15:11:04.185 INFO 7 --- [ main] org.hibernate.dialect.Dialect : HHH000400: Using dialect: org.hibernate.dialect.PostgreSQL10Dialect compreface-admin | 2023-07-15 15:11:04.301 INFO 7 --- [ main] Hibernate Types : This framework is proudly powered by: compreface-admin | compreface-admin | >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> compreface-admin | _ _ _ _ compreface-admin | | | | | (_) | | compreface-admin | | |__| |_ _ _ __ ___ _ __ ___ _ ___| |_ ___ _ __ ___ ___ compreface-admin | | __ | | | | '_ \ / _ \ '__/ __| / __| __/ _ \ '_ \ / __/ _ \ compreface-admin | | | | | |_| | |_) | __/ | \__ \ \__ \ || __/ | | | (_| __/ compreface-admin | |_| |_|\__, | .__/ \___|_| |___/_|___/\__\___|_| |_|\___\___| compreface-admin | __/ | | compreface-admin | |___/|_| compreface-admin | compreface-admin | At Hypersistence, we only build amazing tools, like Hibernate Types, Flexy Pool, or Hypersistence Optimizer. compreface-admin | compreface-admin | What if there were a tool that could automatically detect JPA and Hibernate performance issues? compreface-admin | compreface-admin | Hypersistence Optimizer is that tool! For more details, go to: compreface-admin | compreface-admin | https://vladmihalcea.com/hypersistence-optimizer/ compreface-admin | <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< compreface-admin | compreface-api | 2023-07-15 15:11:04.435 INFO 7 --- [ main] org.hibernate.Version : HHH000412: Hibernate ORM core version 5.4.33 compreface-api | 2023-07-15 15:11:04.442 INFO 7 --- [ main] org.hibernate.cfg.Environment : HHH000205: Loaded properties from resource hibernate.properties: {hibernate.bytecode.use_reflection_optimizer=false, hibernate.types.print.banner=false} compreface-api | 2023-07-15 15:11:05.124 INFO 7 --- [ main] o.hibernate.annotations.common.Version : HCANN000001: Hibernate Commons Annotations {5.1.2.Final} compreface-admin | 2023-07-15 15:11:05.912 INFO 7 --- [ main] o.h.e.t.j.p.i.JtaPlatformInitiator : HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform] compreface-admin | 2023-07-15 15:11:05.928 INFO 7 --- [ main] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default' compreface-api | 2023-07-15 15:11:06.440 INFO 7 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting... compreface-api | 2023-07-15 15:11:06.880 INFO 7 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Start completed. compreface-api | 2023-07-15 15:11:06.991 INFO 7 --- [ main] org.hibernate.dialect.Dialect : HHH000400: Using dialect: org.hibernate.dialect.PostgreSQL10Dialect compreface-admin | 2023-07-15 15:11:08.460 INFO 7 --- [ main] o.s.b.a.e.web.EndpointLinksResolver : Exposing 1 endpoint(s) beneath base path '/actuator' compreface-admin | 2023-07-15 15:11:08.569 INFO 7 --- [ main] pertySourcedRequestMappingHandlerMapping : Mapped URL path [/v2/api-docs] onto method [springfox.documentation.swagger2.web.Swagger2Controller#getDocumentation(String, HttpServletRequest)] compreface-admin | 2023-07-15 15:11:09.215 WARN 7 --- [ main] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/v2/api-docs']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. compreface-admin | 2023-07-15 15:11:09.217 INFO 7 --- [ main] o.s.s.web.DefaultSecurityFilterChain : Will secure Ant [pattern='/v2/api-docs'] with [] compreface-admin | 2023-07-15 15:11:09.217 WARN 7 --- [ main] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/configuration/ui']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. compreface-admin | 2023-07-15 15:11:09.218 INFO 7 --- [ main] o.s.s.web.DefaultSecurityFilterChain : Will secure Ant [pattern='/configuration/ui'] with [] compreface-admin | 2023-07-15 15:11:09.218 WARN 7 --- [ main] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/swagger-resources/**']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. compreface-admin | 2023-07-15 15:11:09.218 INFO 7 --- [ main] o.s.s.web.DefaultSecurityFilterChain : Will secure Ant [pattern='/swagger-resources/**'] with [] compreface-admin | 2023-07-15 15:11:09.218 WARN 7 --- [ main] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/configuration/**']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. compreface-admin | 2023-07-15 15:11:09.218 INFO 7 --- [ main] o.s.s.web.DefaultSecurityFilterChain : Will secure Ant [pattern='/configuration/**'] with [] compreface-admin | 2023-07-15 15:11:09.219 WARN 7 --- [ main] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/swagger-ui**']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. compreface-admin | 2023-07-15 15:11:09.220 INFO 7 --- [ main] o.s.s.web.DefaultSecurityFilterChain : Will secure Ant [pattern='/swagger-ui**'] with [] compreface-admin | 2023-07-15 15:11:09.220 WARN 7 --- [ main] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/webjars/**']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. compreface-admin | 2023-07-15 15:11:09.220 INFO 7 --- [ main] o.s.s.web.DefaultSecurityFilterChain : Will secure Ant [pattern='/webjars/**'] with [] compreface-admin | 2023-07-15 15:11:09.220 WARN 7 --- [ main] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/lms/**']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. compreface-admin | 2023-07-15 15:11:09.220 INFO 7 --- [ main] o.s.s.web.DefaultSecurityFilterChain : Will secure Ant [pattern='/lms/**'] with [] compreface-admin | 2023-07-15 15:11:09.283 INFO 7 --- [ main] o.s.s.web.DefaultSecurityFilterChain : Will not secure Or [Ant [pattern='/oauth/token'], Ant [pattern='/oauth/token_key'], Ant [pattern='/oauth/check_token']] compreface-admin | 2023-07-15 15:11:09.301 INFO 7 --- [ main] o.s.s.web.DefaultSecurityFilterChain : Will not secure org.springframework.security.oauth2.config.annotation.web.configuration.ResourceServerConfiguration$NotOAuthRequestMatcher@28d16af8 compreface-admin | 2023-07-15 15:11:09.306 INFO 7 --- [ main] o.s.s.web.DefaultSecurityFilterChain : Will not secure any request compreface-admin | 2023-07-15 15:11:09.421 INFO 7 --- [ main] org.quartz.impl.StdSchedulerFactory : Using default implementation for ThreadExecutor compreface-admin | 2023-07-15 15:11:09.436 INFO 7 --- [ main] org.quartz.core.SchedulerSignalerImpl : Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl compreface-admin | 2023-07-15 15:11:09.436 INFO 7 --- [ main] org.quartz.core.QuartzScheduler : Quartz Scheduler v.2.3.2 created. compreface-admin | 2023-07-15 15:11:09.439 INFO 7 --- [ main] o.s.s.quartz.LocalDataSourceJobStore : Using db table-based data access locking (synchronization). compreface-admin | 2023-07-15 15:11:09.442 INFO 7 --- [ main] o.s.s.quartz.LocalDataSourceJobStore : JobStoreCMT initialized. compreface-admin | 2023-07-15 15:11:09.442 INFO 7 --- [ main] org.quartz.core.QuartzScheduler : Scheduler meta-data: Quartz Scheduler (v2.3.2) 'statistics scheduler' with instanceId 'NON_CLUSTERED' compreface-admin | Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally. compreface-admin | NOT STARTED. compreface-admin | Currently in standby mode. compreface-admin | Number of jobs executed: 0 compreface-admin | Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 1 threads. compreface-admin | Using job-store 'org.springframework.scheduling.quartz.LocalDataSourceJobStore' - which supports persistence. and is not clustered. compreface-admin | compreface-admin | 2023-07-15 15:11:09.442 INFO 7 --- [ main] org.quartz.impl.StdSchedulerFactory : Quartz scheduler 'statistics scheduler' initialized from an externally provided properties instance. compreface-admin | 2023-07-15 15:11:09.443 INFO 7 --- [ main] org.quartz.impl.StdSchedulerFactory : Quartz scheduler version: 2.3.2 compreface-admin | 2023-07-15 15:11:09.443 INFO 7 --- [ main] org.quartz.core.QuartzScheduler : JobFactory set to: org.springframework.scheduling.quartz.SpringBeanJobFactory@2f82e3cf compreface-admin | 2023-07-15 15:11:09.489 INFO 7 --- [ main] o.s.s.quartz.LocalDataSourceJobStore : Freed 0 triggers from 'acquired' / 'blocked' state. compreface-admin | 2023-07-15 15:11:09.491 INFO 7 --- [ main] o.s.s.quartz.LocalDataSourceJobStore : Recovering 0 jobs that were in-progress at the time of the last shut-down. compreface-admin | 2023-07-15 15:11:09.491 INFO 7 --- [ main] o.s.s.quartz.LocalDataSourceJobStore : Recovery complete. compreface-admin | 2023-07-15 15:11:09.492 INFO 7 --- [ main] o.s.s.quartz.LocalDataSourceJobStore : Removed 0 'complete' triggers. compreface-admin | 2023-07-15 15:11:09.492 INFO 7 --- [ main] o.s.s.quartz.LocalDataSourceJobStore : Removed 0 stale fired job entries. compreface-admin | 2023-07-15 15:11:09.495 INFO 7 --- [ main] org.quartz.core.QuartzScheduler : Scheduler statistics scheduler_$_NON_CLUSTERED started. compreface-admin | 2023-07-15 15:11:10.560 INFO 7 --- [ main] o.a.coyote.http11.Http11NioProtocol : Starting ProtocolHandler ["http-nio-8080"] compreface-admin | 2023-07-15 15:11:10.581 INFO 7 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path '' compreface-admin | 2023-07-15 15:11:10.583 INFO 7 --- [ main] d.s.w.p.DocumentationPluginsBootstrapper : Context refreshed compreface-admin | 2023-07-15 15:11:10.604 INFO 7 --- [ main] d.s.w.p.DocumentationPluginsBootstrapper : Found 1 custom documentation plugin(s) compreface-admin | 2023-07-15 15:11:10.664 INFO 7 --- [ main] s.d.s.w.s.ApiListingReferenceScanner : Scanning for api listing references compreface-admin | 2023-07-15 15:11:10.884 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: regenerateApiKeyUsingPUT_1 compreface-admin | 2023-07-15 15:11:10.966 INFO 7 --- [ main] com.exadel.frs.FrsApplication : Started FrsApplication in 18.459 seconds (JVM running for 20.043) compreface-api | 2023-07-15 15:11:11.008 INFO 7 --- [ main] o.h.e.t.j.p.i.JtaPlatformInitiator : HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform] compreface-api | 2023-07-15 15:11:11.068 INFO 7 --- [ main] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default' compreface-api | 2023-07-15 15:11:17.159 INFO 7 --- [ main] o.s.b.a.e.web.EndpointLinksResolver : Exposing 1 endpoint(s) beneath base path '/actuator' compreface-api | 2023-07-15 15:11:17.433 INFO 7 --- [ main] pertySourcedRequestMappingHandlerMapping : Mapped URL path [/v2/api-docs] onto method [springfox.documentation.swagger2.web.Swagger2Controller#getDocumentation(String, HttpServletRequest)] compreface-api | 2023-07-15 15:11:17.511 INFO 7 --- [ main] org.quartz.impl.StdSchedulerFactory : Using default implementation for ThreadExecutor compreface-api | 2023-07-15 15:11:17.533 INFO 7 --- [ main] org.quartz.core.SchedulerSignalerImpl : Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl compreface-api | 2023-07-15 15:11:17.533 INFO 7 --- [ main] org.quartz.core.QuartzScheduler : Quartz Scheduler v.2.3.2 created. compreface-api | 2023-07-15 15:11:17.534 INFO 7 --- [ main] org.quartz.simpl.RAMJobStore : RAMJobStore initialized. compreface-api | 2023-07-15 15:11:17.536 INFO 7 --- [ main] org.quartz.core.QuartzScheduler : Scheduler meta-data: Quartz Scheduler (v2.3.2) 'quartzScheduler' with instanceId 'NON_CLUSTERED' compreface-api | Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally. compreface-api | NOT STARTED. compreface-api | Currently in standby mode. compreface-api | Number of jobs executed: 0 compreface-api | Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 10 threads. compreface-api | Using job-store 'org.quartz.simpl.RAMJobStore' - which does not support persistence. and is not clustered. compreface-api | compreface-api | 2023-07-15 15:11:17.536 INFO 7 --- [ main] org.quartz.impl.StdSchedulerFactory : Quartz scheduler 'quartzScheduler' initialized from an externally provided properties instance. compreface-api | 2023-07-15 15:11:17.536 INFO 7 --- [ main] org.quartz.impl.StdSchedulerFactory : Quartz scheduler version: 2.3.2 compreface-api | 2023-07-15 15:11:17.537 INFO 7 --- [ main] org.quartz.core.QuartzScheduler : JobFactory set to: org.springframework.scheduling.quartz.SpringBeanJobFactory@73da303e compreface-api | 2023-07-15 15:11:17.544 INFO 7 --- [ main] org.quartz.core.QuartzScheduler : Scheduler quartzScheduler_$_NON_CLUSTERED started. compreface-api | 2023-07-15 15:11:18.466 INFO 7 --- [ main] o.a.coyote.http11.Http11NioProtocol : Starting ProtocolHandler ["http-nio-8080"] compreface-api | 2023-07-15 15:11:18.485 INFO 7 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path '' compreface-api | 2023-07-15 15:11:18.487 INFO 7 --- [ main] d.s.w.p.DocumentationPluginsBootstrapper : Context refreshed compreface-api | 2023-07-15 15:11:18.557 INFO 7 --- [ main] d.s.w.p.DocumentationPluginsBootstrapper : Found 2 custom documentation plugin(s) compreface-api | 2023-07-15 15:11:18.752 INFO 7 --- [ main] s.d.s.w.s.ApiListingReferenceScanner : Scanning for api listing references compreface-api | 2023-07-15 15:11:19.293 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: recognizeBase64UsingPOST_1 compreface-api | 2023-07-15 15:11:19.295 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: downloadImgUsingGET_1 compreface-api | 2023-07-15 15:11:19.379 INFO 7 --- [ main] s.d.s.w.s.ApiListingReferenceScanner : Scanning for api listing references compreface-api | 2023-07-15 15:11:19.402 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: getConfigUsingGET_1 compreface-api | 2023-07-15 15:11:19.405 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: getCheckDemoUsingGET_1 compreface-api | 2023-07-15 15:11:19.420 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: detectUsingPOST_1 compreface-api | 2023-07-15 15:11:19.425 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: detectBase64UsingPOST_1 compreface-api | 2023-07-15 15:11:19.430 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: addEmbeddingUsingPOST_1 compreface-api | 2023-07-15 15:11:19.433 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: addEmbeddingBase64UsingPOST_1 compreface-api | 2023-07-15 15:11:19.435 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: deleteEmbeddingByIdUsingDELETE_1 compreface-api | 2023-07-15 15:11:19.439 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: deleteEmbeddingsByIdUsingPOST_1 compreface-api | 2023-07-15 15:11:19.441 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: downloadImgUsingGET_2 compreface-api | 2023-07-15 15:11:19.446 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: listEmbeddingsUsingGET_1 compreface-api | 2023-07-15 15:11:19.459 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: recognizeBase64UsingPOST_2 compreface-api | 2023-07-15 15:11:19.463 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: recognizeFileUsingPOST_1 compreface-api | 2023-07-15 15:11:19.465 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: removeAllSubjectEmbeddingsUsingDELETE_1 compreface-api | 2023-07-15 15:11:19.467 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: migrateUsingPOST_1 compreface-api | 2023-07-15 15:11:19.472 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: handleUsingGET_1 compreface-api | 2023-07-15 15:11:19.483 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: recognizeUsingPOST_1 compreface-api | 2023-07-15 15:11:19.486 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: recognizeBase64UsingPOST_3 compreface-api | 2023-07-15 15:11:19.488 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: downloadImgUsingGET_3 compreface-api | 2023-07-15 15:11:19.492 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: createSubjectUsingPOST_1 compreface-api | 2023-07-15 15:11:19.494 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: deleteSubjectUsingDELETE_1 compreface-api | 2023-07-15 15:11:19.495 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: deleteSubjectsUsingDELETE_1 compreface-api | 2023-07-15 15:11:19.497 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: listSubjectsUsingGET_1 compreface-api | 2023-07-15 15:11:19.499 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: renameSubjectUsingPUT_1 compreface-api | 2023-07-15 15:11:19.513 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: verifyUsingPOST_1 compreface-api | 2023-07-15 15:11:19.516 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: verifyBase64UsingPOST_1 compreface-api | 2023-07-15 15:11:19.552 INFO 7 --- [ main] com.exadel.frs.TrainServiceApplication : Started TrainServiceApplication in 27.435 seconds (JVM running for 28.993)

[pospielov]

I don't see any errors in logs you provided.
Could you add logs where you see "failed call to cuInit: UNKNOWN ERROR"?
Also, please attach logs in a text file next time

[ishangxue]

2023-12-20 14:12:46 [uWSGI] getting INI configuration from uwsgi.ini
2023-12-20 14:12:46 *** Starting uWSGI 2.0.19 (64bit) on [Wed Dec 20 06:12:46 2023] ***
2023-12-20 14:12:46 compiled with version: 10.2.1 20210110 on 14 August 2023 20:24:09
2023-12-20 14:12:46 os: Linux-5.15.133.1-microsoft-standard-WSL2 #1 SMP Thu Oct 5 21:02:42 UTC 2023
2023-12-20 14:12:46 nodename: 1d2ec035cb59
2023-12-20 14:12:46 machine: x86_64
2023-12-20 14:12:46 clock source: unix
2023-12-20 14:12:46 pcre jit disabled
2023-12-20 14:12:46 detected number of CPU cores: 8
2023-12-20 14:12:46 current working directory: /app/ml
2023-12-20 14:12:46 detected binary path: /usr/local/bin/uwsgi
2023-12-20 14:12:46 setgid() to 33
2023-12-20 14:12:46 setuid() to 33
2023-12-20 14:12:46 your memory page size is 4096 bytes
2023-12-20 14:12:46 detected max file descriptor number: 1048576
2023-12-20 14:12:46 lock engine: pthread robust mutexes
2023-12-20 14:12:46 thunder lock: disabled (you can enable it with --thunder-lock)
2023-12-20 14:12:46 uwsgi socket 0 bound to TCP address 0.0.0.0:3000 fd 3
2023-12-20 14:12:46 Python version: 3.8.17 (default, Jul 28 2023, 06:10:22) [GCC 10.2.1 20210110]
2023-12-20 14:12:46 *** Python threads support is disabled. You can enable it with --enable-threads ***
2023-12-20 14:12:46 Python main interpreter initialized at 0x5647850126c0
2023-12-20 14:12:46 your server socket listen backlog is limited to 100 connections
2023-12-20 14:12:46 your mercy for graceful operations on workers is 60 seconds
2023-12-20 14:12:46 mapped 145808 bytes (142 KB) for 1 cores
2023-12-20 14:12:46 *** Operational MODE: single process ***
2023-12-20 14:12:48 {"severity": "INFO", "message": "{'BUILD_VERSION': 'dev', 'CALCULATION_PLUGIN': 'facenet.Calculator', 'EXTRA_PLUGINS': ['facenet.LandmarksDetector', 'agegender.AgeDetector', 'agegender.GenderDetector', 'facenet.facemask.MaskDetector', 'facenet.PoseEstimator'], 'FACE_DETECTION_PLUGIN': 'facenet.FaceDetector', 'GPU_IDX': -1, 'IMG_LENGTH_LIMIT': 640, 'INTEL_OPTIMIZATION': False, 'IS_DEV_ENV': False, 'LOGGING_LEVEL_NAME': 'DEBUG', 'ML_PORT': 3000, 'RUN_MODE': True}", "request": null, "logger": "src.app", "module": "app", "traceback": null, "build_version": "dev"}
2023-12-20 14:12:48 {"severity": "DEBUG", "message": "Creating new app for WSGI", "request": null, "logger": "src.app", "module": "app", "traceback": null, "build_version": "dev"}
2023-12-20 14:12:48 WSGI app 0 (mountpoint='') ready in 2 seconds on interpreter 0x5647850126c0 pid: 1 (default app)
2023-12-20 14:12:48 *** uWSGI is running in multiple interpreter mode ***
2023-12-20 14:12:48 spawned uWSGI master process (pid: 1)
2023-12-20 14:12:48 spawned uWSGI worker 1 (pid: 22, cores: 1)
2023-12-20 14:14:11 2023-12-20 06:14:11.090500: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: UNKNOWN ERROR (303)
2023-12-20 14:14:11 {"severity": "DEBUG", "message": "Found: BoundingBoxDTO(x_min=49, y_min=47, x_max=199, y_max=224, probability=0.9400066137313843, _np_landmarks=array([[104, 124],\n [153, 124],\n [131, 157],\n [106, 177],\n [146, 177]]))", "request": {"method": "GET", "path": "/", "filename": "", "api_key": "", "remote_addr": "172.17.0.1"}, "logger": "src.services.facescan.plugins.facenet.facenet", "module": "facenet", "traceback": null, "build_version": "dev"}
2023-12-20 14:14:59 Starting to load ML models
2023-12-20 14:14:59 {"severity": "WARNING", "message": "404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.", "request": {"method": "GET", "path": "/", "filename": "", "api_key": "", "remote_addr": "172.17.0.1"}, "logger": "root", "module": "error_handling", "traceback": "Traceback (most recent call last):\n File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request\n rv = self.dispatch_request()\n File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1926, in dispatch_request\n self.raise_routing_exception(req)\n File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1908, in raise_routing_exception\n raise request.routing_exception\n File "/usr/local/lib/python3.8/site-packages/flask/ctx.py", line 350, in match_request\n result = self.url_adapter.match(return_rule=True)\n File "/usr/local/lib/python3.8/site-packages/werkzeug/routing.py", line 1945, in match\n raise NotFound()\nwerkzeug.exceptions.NotFound: 404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.\n", "build_version": "dev"}

[ishangxue]

I don't see any errors in logs you provided. Could you add logs where you see "failed call to cuInit: UNKNOWN ERROR"? Also, please attach logs in a text file next time

2023-12-20 14:12:46 [uWSGI] getting INI configuration from uwsgi.ini
2023-12-20 14:12:46 *** Starting uWSGI 2.0.19 (64bit) on [Wed Dec 20 06:12:46 2023] ***
2023-12-20 14:12:46 compiled with version: 10.2.1 20210110 on 14 August 2023 20:24:09
2023-12-20 14:12:46 os: Linux-5.15.133.1-microsoft-standard-WSL2 #1 SMP Thu Oct 5 21:02:42 UTC 2023
2023-12-20 14:12:46 nodename: 1d2ec035cb59
2023-12-20 14:12:46 machine: x86_64
2023-12-20 14:12:46 clock source: unix
2023-12-20 14:12:46 pcre jit disabled
2023-12-20 14:12:46 detected number of CPU cores: 8
2023-12-20 14:12:46 current working directory: /app/ml
2023-12-20 14:12:46 detected binary path: /usr/local/bin/uwsgi
2023-12-20 14:12:46 setgid() to 33
2023-12-20 14:12:46 setuid() to 33
2023-12-20 14:12:46 your memory page size is 4096 bytes
2023-12-20 14:12:46 detected max file descriptor number: 1048576
2023-12-20 14:12:46 lock engine: pthread robust mutexes
2023-12-20 14:12:46 thunder lock: disabled (you can enable it with --thunder-lock)
2023-12-20 14:12:46 uwsgi socket 0 bound to TCP address 0.0.0.0:3000 fd 3
2023-12-20 14:12:46 Python version: 3.8.17 (default, Jul 28 2023, 06:10:22) [GCC 10.2.1 20210110]
2023-12-20 14:12:46 *** Python threads support is disabled. You can enable it with --enable-threads ***
2023-12-20 14:12:46 Python main interpreter initialized at 0x5647850126c0
2023-12-20 14:12:46 your server socket listen backlog is limited to 100 connections
2023-12-20 14:12:46 your mercy for graceful operations on workers is 60 seconds
2023-12-20 14:12:46 mapped 145808 bytes (142 KB) for 1 cores
2023-12-20 14:12:46 *** Operational MODE: single process ***
2023-12-20 14:12:48 {"severity": "INFO", "message": "{'BUILD_VERSION': 'dev', 'CALCULATION_PLUGIN': 'facenet.Calculator', 'EXTRA_PLUGINS': ['facenet.LandmarksDetector', 'agegender.AgeDetector', 'agegender.GenderDetector', 'facenet.facemask.MaskDetector', 'facenet.PoseEstimator'], 'FACE_DETECTION_PLUGIN': 'facenet.FaceDetector', 'GPU_IDX': -1, 'IMG_LENGTH_LIMIT': 640, 'INTEL_OPTIMIZATION': False, 'IS_DEV_ENV': False, 'LOGGING_LEVEL_NAME': 'DEBUG', 'ML_PORT': 3000, 'RUN_MODE': True}", "request": null, "logger": "src.app", "module": "app", "traceback": null, "build_version": "dev"}
2023-12-20 14:12:48 {"severity": "DEBUG", "message": "Creating new app for WSGI", "request": null, "logger": "src.app", "module": "app", "traceback": null, "build_version": "dev"}
2023-12-20 14:12:48 WSGI app 0 (mountpoint='') ready in 2 seconds on interpreter 0x5647850126c0 pid: 1 (default app)
2023-12-20 14:12:48 *** uWSGI is running in multiple interpreter mode ***
2023-12-20 14:12:48 spawned uWSGI master process (pid: 1)
2023-12-20 14:12:48 spawned uWSGI worker 1 (pid: 22, cores: 1)
2023-12-20 14:14:11 2023-12-20 06:14:11.090500: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: UNKNOWN ERROR (303)
2023-12-20 14:14:11 {"severity": "DEBUG", "message": "Found: BoundingBoxDTO(x_min=49, y_min=47, x_max=199, y_max=224, probability=0.9400066137313843, _np_landmarks=array([[104, 124],\n [153, 124],\n [131, 157],\n [106, 177],\n [146, 177]]))", "request": {"method": "GET", "path": "/", "filename": "", "api_key": "", "remote_addr": "172.17.0.1"}, "logger": "src.services.facescan.plugins.facenet.facenet", "module": "facenet", "traceback": null, "build_version": "dev"}
2023-12-20 14:14:59 Starting to load ML models
2023-12-20 14:14:59 {"severity": "WARNING", "message": "404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.", "request": {"method": "GET", "path": "/", "filename": "", "api_key": "", "remote_addr": "172.17.0.1"}, "logger": "root", "module": "error_handling", "traceback": "Traceback (most recent call last):\n File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request\n rv = self.dispatch_request()\n File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1926, in dispatch_request\n self.raise_routing_exception(req)\n File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1908, in raise_routing_exception\n raise request.routing_exception\n File "/usr/local/lib/python3.8/site-packages/flask/ctx.py", line 350, in match_request\n result = self.url_adapter.match(return_rule=True)\n File "/usr/local/lib/python3.8/site-packages/werkzeug/routing.py", line 1945, in match\n raise NotFound()\nwerkzeug.exceptions.NotFound: 404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.\n", "build_version": "dev"}

[ishangxue]

Hello, my computer does not have a GPU, I want to use CPU, but in Docker desktop after booting e Tensorflow/stream/Cuda/Cuda. CC: 313] failed call to cuInit: UNKNOWN ERROR (303) , I think this is because I don't have a GPU problem with my graphics card, now I want to use CPU OK? How Do I start it if I can?

[pospielov]

This error is OK.
Please post your configuration here: #1216