Add one liner documentation how to actually use CI mode

[janknips]

I can't figure out how to use it, because there used to be a flag for it, which is now gone, and the entire project contains no mention of the word CI anymore :D

[welcome]

Hi! Thanks for opening your first issue here! 😄

[fzyzcjy]

You are right, I forgot to update the doc...

[fzyzcjy]

This is how I personally use it:

dart run convenient_test_manager_dart # in one shell
flutter run path/to/your/integration_test/main_test.dart --host-vmservice-port 9753 --disable-service-auth-codes --dart-define CONVENIENT_TEST_APP_CODE_DIR=sth # in another shell, run your worker app

[janknips]

But will it terminate like that? Seems like this is what I'm already doing locally 😁 So for CI I would need a mode that terminates after all tests ran

[fzyzcjy]

Yes it terminates automatically after all tests run.

[janknips]

This is how I personally use it:

dart run convenient_test_manager_dart # in one shell
flutter run path/to/your/integration_test/main_test.dart --host-vmservice-port 9753 --disable-service-auth-codes --dart-define CONVENIENT_TEST_APP_CODE_DIR=sth # in another shell, run your worker app

Could not find file convenient_test_manager 😛 Which directory are you running in / which build command did you run before this?

[fzyzcjy]

Could not find file convenient_test_manager

https://github.com/fzyzcjy/flutter_convenient_test/tree/master/packages/convenient_test_manager_dart

[janknips]

Thanks, got it to run. Unfortunately, it seems quite unreliable, e.g. worker disconnects, etc, at least I never got my tests to pass, which usually easily pass.

Maybe it would be good to add another barebone CI mode that will just detect wether flutter test was used, and in that case, just work as if only integration_test was used, without using the manager. What do you think? At least in my use case I can imagine just calling this in CI, since it's fast, and failed tests could still be re-run locally afterwards.

Logs:

2022-08-01T13:40:56.402721Z|info|VmServiceWrapperService|Connecting to vm service at ws://127.0.0.1:9753/ws. Please ensure your Flutter app has port=9753
2022-08-01T13:40:56.404170Z|warn|VmServiceWrapperService|init failed e=SocketException: Connection refused (OS Error: Connection refused, errno = 61), address = 127.0.0.1, port = 63525 s=#0      _NativeSocket.startConnect (dart:io-patch/socket_patch.dart:682:35)[NL]#1      _RawSocket.startConnect (dart:io-patch/socket_patch.dart:1817:26)[NL]#2      RawSocket.startConnect (dart:io-patch/socket_patch.dart:27:23)[NL]#3      Socket._startConnect (dart:io-patch/socket_patch.dart:2038:22)[NL]#4      Socket.startConnect (dart:io/socket.dart:759:21)[NL]#5      _ConnectionTarget.connect (dart:_http/http_impl.dart:2453:20)[NL]#6      _HttpClient._getConnection.connect (dart:_http/http_impl.dart:2867:12)[NL]#7      _HttpClient._getConnection (dart:_http/http_impl.dart:2872:12)[NL]#8      _HttpClient.
_openUrl (dart:_http/http_impl.dart:2727:12)[NL]#9      _HttpClient.openUrl (dart:_http/http_impl.dart:2591:7)[NL]#10     _WebSocketImpl.connect (dart:_http/websocket_impl.dart:1021:42)[NL]#11     WebSocket.connect (dart:_http/websocket.dart:360:22)[NL]#12     vmServiceConnectUri (package:vm_service/vm_service_io.dart:32:44)[NL]#13     VmServiceWrapperService.connect (package:convenient_test_manager_dart/services/vm_service_wrapper_service.dart:30:31)[NL]#14     _awaitWorkerAvailable (file:///Users/jan/StudioProjects/flutter_convenient_test/packages/convenient_test_manager_dart/bin/convenient_test_manager_dart.dart:63:37)[NL]<asynchronous suspension>[NL]#15     main (file:///Users/jan/StudioProjects/flutter_convenient_test/packages/convenient_test_manager_dart/bin/convenient_test_manager_d
art.dart:25:3)[NL]<asynchronous suspension>[NL]
2022-08-01T13:40:57.387018Z|debug|ConvenientTestManagerService|getWorkerCurrentRunConfig ans=integrationTest: {[NL]  reportSuiteInfo: true[NL]  executionFilter: {[NL]    filterNameRegex: match-nothing^$[NL]    strategy: {[NL]      allMatch: {[NL]      }[NL]    }[NL]  }[NL]  defaultRetryCount: 1[NL]  autoUpdateGoldenFiles: false[NL]}[NL] currSuperRunController=seenTearDownAll: false[NL]    
2022-08-01T13:40:57.650498Z|debug|ReportHandlerService|SetUpAll 
2022-08-01T13:40:57.675875Z|debug|VideoRecorderStore|startRecord call ScreenVideoRecorderService begin
2022-08-01T13:40:57.678795Z|info|ScreenVideoRecorderServiceIosSimulator|startRecord begin /tmp/ConvenientTest/RUN-20220801-033838-263/Video/20220801_034057.mov
2022-08-01T13:40:57.689800Z|debug|ReportHandlerService|handleReportSuiteInfo called groupId: 1[NL]groups: {[NL]  id: 570286508[NL]  name: Sign in tests:[NL]  parentId: 1[NL]  entryIds: 34833574[NL]  entryIds: 210372701[NL]}[NL]groups: {[NL]  id: 1[NL]  name: [NL]  parentId: -1[NL]  entryIds: 570286508[NL]}[NL]tests: {[NL]  id: 34833574[NL]  name: Sign in tests: can sign in as feedFollower[NL]  parentId: 570286508[NL]}[NL]tests: {[NL]  id: 210372701[NL]  name: Sign in tests: can sign in as contributor[NL]  parentId: 570286508[NL]}[NL]
2022-08-01T13:40:57.690148Z|debug|ReportHandlerService|handleReportSuiteInfo thus clearAll
2022-08-01T13:40:57.690500Z|debug|MiscDartService|clearAll
2022-08-01T13:40:57.693762Z|debug|ReportHandlerService|handleReportSuiteInfo thus clear
2022-08-01T13:40:57.694380Z|debug|ReportSaverService|clear
2022-08-01T13:40:57.719333Z|debug|ReportHandlerService|StateChange: testName=(tearDownAll) state=status: running[NL]result: success[NL]
2022-08-01T13:40:57.732418Z|debug|ReportHandlerService|Message: Warning: integration_test plugin was not detected.
2022-08-01T13:40:57.732796Z|debug|ReportHandlerService|Message: 
2022-08-01T13:40:57.733438Z|debug|ReportHandlerService|Message: If you're running the tests with `flutter drive`, please make sure your tests
2022-08-01T13:40:57.734213Z|debug|ReportHandlerService|Message: are in the `integration_test/` directory of your package and use
2022-08-01T13:40:57.734957Z|debug|ReportHandlerService|Message: `flutter test $path_to_test` to run it instead.
2022-08-01T13:40:57.736262Z|debug|ReportHandlerService|Message: 
2022-08-01T13:40:57.738320Z|debug|ReportHandlerService|Message: If you're running the tests with Android instrumentation or XCTest, this means
2022-08-01T13:40:57.741175Z|debug|ReportHandlerService|Message: that you are not capturing test results properly! See the following link for
2022-08-01T13:40:57.742487Z|debug|ReportHandlerService|Message: how to set up the integration_test plugin:
2022-08-01T13:40:57.743804Z|debug|ReportHandlerService|Message: 
2022-08-01T13:40:57.745787Z|debug|ReportHandlerService|Message: https://flutter.dev/docs/testing/integration-tests#testing-on-firebase-test-lab
2022-08-01T13:40:57.747829Z|debug|ReportHandlerService|Message: 
2022-08-01T13:40:57.752263Z|debug|ReportHandlerService|TearDownAll resolvedExecutionFilter: {[NL]}[NL]
2022-08-01T13:40:57.753437Z|info|VideoRecorderStore|stopRecord skip since recordingVideoInfo==null
2022-08-01T13:40:57.795430Z|debug|VideoRecorderStore|startRecord call ScreenVideoRecorderService end
2022-08-01T13:40:57.802530Z|debug|ReportHandlerService|StateChange: testName=(tearDownAll) state=status: complete[NL]result: success[NL]
2022-08-01T13:40:58.223250Z|debug|ScreenVideoRecorderServiceIosSimulator|[STDERR] Recording started[NL]
2022-08-01T13:40:59.406865Z|info|main|waitWorkerAvailable check
2022-08-01T13:40:59.407203Z|info|VmServiceWrapperService|Connecting to vm service at ws://127.0.0.1:9753/ws. Please ensure your Flutter app has port=9753
2022-08-01T13:40:59.464867Z|info|_ServiceConnectionManager|vmServiceOpened
2022-08-01T13:40:59.501701Z|info|_ServiceConnectionManager|handleServiceEvent kind=ServiceRegistered service=reloadSources method=s0.reloadSources
2022-08-01T13:40:59.503067Z|info|_ServiceConnectionManager|handleServiceEvent kind=ServiceRegistered service=hotRestart method=s0.hotRestart
2022-08-01T13:40:59.503176Z|info|_ServiceConnectionManager|handleServiceEvent kind=ServiceRegistered service=flutterVersion method=s0.flutterVersion
2022-08-01T13:40:59.503272Z|info|_ServiceConnectionManager|handleServiceEvent kind=ServiceRegistered service=compileExpression method=s0.compileExpression
2022-08-01T13:40:59.503369Z|info|_ServiceConnectionManager|handleServiceEvent kind=ServiceRegistered service=flutterMemoryInfo method=s0.flutterMemoryInfo
2022-08-01T13:40:59.503465Z|info|_ServiceConnectionManager|handleServiceEvent kind=ServiceRegistered service=flutterGetSkSL method=s0.flutterGetSkSL
2022-08-01T13:41:02.576583Z|info|main|waitWorkerAvailable check
2022-08-01T13:41:02.578734Z|info|main|monitorWorkerAvailable check
2022-08-01T13:41:02.578871Z|info|main|step extra sleep to avoid too quickly hot-restart worker
2022-08-01T13:41:07.582168Z|info|main|monitorWorkerAvailable check
2022-08-01T13:41:08.581594Z|info|main|step reloadInfo
2022-08-01T13:41:08.583016Z|info|VmServiceWrapperService|hotRestartThrottled triggered
2022-08-01T13:41:08.584786Z|debug|SingleRunningExecutor|call runner() reason= triggerTime=2022-08-01 15:41:08.583506 arg=null
2022-08-01T13:41:08.587864Z|info|_ServiceConnectionManager|hotRestart start
2022-08-01T13:41:08.589244Z|info|main|awaitSuiteInfoNonEmpty check numGroupEntries=4
2022-08-01T13:41:08.590753Z|info|main|step hotRestartAndRunTests
2022-08-01T13:41:08.591026Z|debug|MiscDartService|hotRestartAndRunTests filterNameRegex=.*
2022-08-01T13:41:08.591200Z|info|VmServiceWrapperService|hotRestartThrottled triggered
2022-08-01T13:41:08.591934Z|info|main|step awaitSuperRunStatusTestAllDone
2022-08-01T13:41:09.798020Z|info|_ServiceConnectionManager|hotRestart end resp={type: Success}
2022-08-01T13:41:09.798168Z|debug|VmServiceWrapperService|hotRestartThrottledExecutor deliberately extra wait
2022-08-01T13:41:10.014816Z|debug|ConvenientTestManagerService|getWorkerCurrentRunConfig ans=integrationTest: {[NL]  reportSuiteInfo: true[NL]  executionFilter: {[NL]    filterNameRegex: .*[NL]    strategy: {[NL]      allMatch: {[NL]      }[NL]    }[NL]  }[NL]  defaultRetryCount: 1[NL]  autoUpdateGoldenFiles: false[NL]}[NL] currSuperRunController=seenTearDownAll: false[NL]    
2022-08-01T13:41:10.233986Z|debug|ReportHandlerService|SetUpAll 
2022-08-01T13:41:10.235581Z|debug|VideoRecorderStore|startRecord call ScreenVideoRecorderService begin
2022-08-01T13:41:10.235676Z|info|ScreenVideoRecorderServiceIosSimulator|startRecord begin /tmp/ConvenientTest/RUN-20220801-034108-587/Video/20220801_034110.mov
2022-08-01T13:41:10.236413Z|info|ScreenVideoRecorderServiceIosSimulator|stopRecord begin
2022-08-01T13:41:10.236476Z|info|ScreenVideoRecorderServiceIosSimulator|stopRecord send signals
2022-08-01T13:41:10.237278Z|info|ScreenVideoRecorderServiceIosSimulator|stopRecord await exitCode
2022-08-01T13:41:10.238181Z|debug|ReportHandlerService|handleReportSuiteInfo called groupId: 1[NL]groups: {[NL]  id: 570286508[NL]  name: Sign in tests:[NL]  parentId: 1[NL]  entryIds: 34833574[NL]  entryIds: 210372701[NL]}[NL]groups: {[NL]  id: 1[NL]  name: [NL]  parentId: -1[NL]  entryIds: 570286508[NL]}[NL]tests: {[NL]  id: 34833574[NL]  name: Sign in tests: can sign in as feedFollower[NL]  parentId: 570286508[NL]}[NL]tests: {[NL]  id: 210372701[NL]  name: Sign in tests: can sign in as contributor[NL]  parentId: 570286508[NL]}[NL]
2022-08-01T13:41:10.238306Z|debug|ReportHandlerService|handleReportSuiteInfo thus clearAll
2022-08-01T13:41:10.238353Z|debug|MiscDartService|clearAll
2022-08-01T13:41:10.238468Z|debug|ReportHandlerService|handleReportSuiteInfo thus clear
2022-08-01T13:41:10.238517Z|debug|ReportSaverService|clear
2022-08-01T13:41:10.240339Z|debug|ReportHandlerService|StateChange: testName=Sign in tests: can sign in as feedFollower state=status: running[NL]result: success[NL]
2022-08-01T13:41:10.380292Z|debug|ReportHandlerService|Message: 2022-08-01T13:41:10.257496Z|info|LogHandle|🟤 START  null null
2022-08-01T13:41:10.383371Z|debug|ReportHandlerService|handleReportLogEntry called
🚀 [STATUS] [0:02] [1x pending, 1x running] [Test=Sign in tests: can sign in as feedFollower]
2022-08-01T13:41:10.657689Z|debug|ReportHandlerService|Message: 2022-08-01T13:41:10.599764Z|info|LogHandle|🔵 START APP  null null
2022-08-01T13:41:10.663036Z|debug|ReportHandlerService|handleReportLogEntry called
2022-08-01T13:41:11.729195Z|info|_ServiceConnectionManager|VMService.onDone called
2022-08-01T13:41:11.869360Z|debug|ScreenVideoRecorderServiceIosSimulator|[STDERR] Wrote video to: /tmp/ConvenientTest/RUN-20220801-033838-263/Video/20220801_034057.mov[NL]
2022-08-01T13:41:11.877720Z|info|ScreenVideoRecorderServiceIosSimulator|stopRecord exitCode=0
2022-08-01T13:41:11.886680Z|debug|VideoRecorderStore|startRecord call ScreenVideoRecorderService end
2022-08-01T13:41:12.239680Z|debug|ScreenVideoRecorderServiceIosSimulator|[STDERR] Recording started[NL]
2022-08-01T13:41:12.583738Z|info|main|monitorWorkerAvailable check
2022-08-01T13:41:12.584165Z|error|main|monitorWorkerAvailable see hot restart not available, thus exit with code=2 (vmServiceWrapperService.hotRestartAvailable=false, vmServiceWrapperService.connected=false)

and the runner:

flutter: 00:00 +0: Sign in tests: can sign in as feedFollower
flutter: 2022-08-01T13:41:10.257496Z|info|LogHandle|🟤 START  null null
flutter: 2022-08-01T13:41:10.599764Z|info|LogHandle|🔵 START APP  null null
Lost connection to device.

[fzyzcjy]

Lost connection to device.

Strongly suspect it is because: #188. See that issue for a lot of details.

It is discussed in: flutter/flutter#37818

And it is fixed: flutter/engine#34496, but not landed in stable yet.

[janknips]

Lost connection to device.

Strongly suspect it is because: #188. See that issue for a lot of details.

It is discussed in: flutter/flutter#37818

And it is fixed: flutter/engine#34496, but not landed in stable.

I see, thanks 👍 I will try again with master branch and let you know. Let's hope for a either hotfix or fast next update 😅

[fzyzcjy]

Unfortunately, it seems quite unreliable, e.g. worker disconnects, etc, at least I never got my tests to pass, which usually easily pass.

If the only problem is "lost connection to device" maybe it is #188 problem; if many other problems feel free to create separate issues

[fzyzcjy]

I see, thanks 👍 I will try again with master branch and let you know. Let's hope for a either hotfix or fast next update 😅

Yes personally I also suffer from this problem :( So eagerly looking forward to next stable!

[janknips]

Unfortunately, it seems quite unreliable, e.g. worker disconnects, etc, at least I never got my tests to pass, which usually easily pass.

If the only problem is "lost connection to device" maybe it is #188 problem; if many other problems feel free to create separate issues

Thanks, yeah for now I can't really tell if anything else is broken 😄 will let you know if I find any other issues!

[fzyzcjy]

Maybe it would be good to add another barebone CI mode that will just detect wether flutter test was used, and in that case, just work as if only integration_test was used, without using the manager. What do you think? At least in my use case I can imagine just calling this in CI, since it's fast, and failed tests could still be re-run locally afterwards.

Some features will be lost (e.g. "isolation mode", "all screenshots and videos", etc). But I guess it is implement-able. Feel free to PR!

[fzyzcjy]

Thanks, yeah for now I can't really tell if anything else is broken 😄 will let you know if I find any other issues!

Sure, take your time! I personally feel it stable except for that problem; for my own case it is sometimes also not stable because I did e2e tests with a backend and docker etc which is source of unstableness - but that is surely not your case and not related to this package

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new issue.