This repository has been archived by the owner on Jun 21, 2018. It is now read-only.
forked from churchers/vm-bhyve
-
Notifications
You must be signed in to change notification settings - Fork 1
/
vm.8
1391 lines (1391 loc) · 45.4 KB
/
vm.8
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
.Dd November 16, 2016
.Dt VM-BHYVE 8
.Os
.Sh NAME
.Nm vm
.Nd "utility to manage bhyve virtual machines"
.Nd v1.2
.Sh SYNOPSIS
.Nm
.Cm version
.Nm
.Cm init
.Pp
.Nm
.Cm get
.Ar all
.Op Ar setting
.Op Ar ...
.Nm
.Cm set
.Ar setting=value
.Op Ar ...
.Pp
.Nm
.Cm switch list
.Nm
.Cm switch info
.Op Ar name
.Op Ar ...
.Nm
.Cm switch create
.Op Fl t Ar type
.Op Fl i Ar interface
.Op Fl n Ar vlan-id
.Op Fl b Ar bridge
.Op Fl m Ar mtu
.Op Fl a Ar address
.Op Fl p
.Ar name
.Nm
.Cm switch vlan
.Ar name vlan-id
.Nm
.Cm switch nat
.Ar name on|off
.Nm
.Cm switch private
.Ar name on|off
.Nm
.Cm switch add
.Ar name interface
.Nm
.Cm switch remove
.Ar name interface
.Nm
.Cm switch destroy
.Ar name
.Pp
.Nm
.Cm datastore list
.Nm
.Cm datastore add
.Ar name spec
.Nm
.Cm datastore remove
.Ar name
.Nm
.Cm datastore iso
.Ar name path
.Pp
.Nm
.Cm create
.Op Fl d Ar datastore
.Op Fl t Ar template
.Op Fl s Ar size
.Ar name
.Nm
.Cm destroy
.Ar name
.Nm
.Cm list
.Nm
.Cm info
.Op Ar name
.Op Ar ...
.Nm
.Op Fl fi
.Cm install
.Ar name iso
.Nm
.Op Fl fi
.Cm start
.Ar name
.Ar ...
.Nm
.Cm stop
.Ar name
.Ar ...
.Nm
.Cm console
.Ar name
.Op Ar com1|com2
.Nm
.Cm rename
.Ar name
.Ar new-name
.Nm
.Cm add
.Op Fl d Ar device
.Op Fl t Ar type
.Op Fl s Ar size|switch
.Ar name
.Nm
.Cm reset
.Ar name
.Nm
.Cm poweroff
.Ar name
.Nm
.Cm startall
.Nm
.Cm stopall
.Nm
.Cm configure
.Ar name
.Nm
.Cm passthru
.Nm
.Cm clone
.Ar name[@snapshot]
.Ar new-name
.Nm
.Cm snapshot
.Op Fl f
.Ar name|name@snapshot
.Nm
.Cm rollback
.Op Fl r
.Ar name@snapshot
.Nm
.Cm send
.Op Fl 12t
.Op Fl i Ar snapshot
.Ar name
.Ar host:port
.Nm
.Cm recv
.Op Fl 12ts
.Op Fl d Ar datastore
.Ar name
.Pp
.Nm
.Cm iso
.Op Ar url
.Pp
.Nm
.Cm image list
.Nm
.Cm image create
.Op Fl d Ar description
.Op Fl u
.Ar name
.Nm
.Cm image provision
.Op Fl d Ar datastore
.Ar uuid
.Ar new-name
.Nm
.Cm image destroy
.Ar uuid
.\" ============ DESCRIPTION =============
.Sh DESCRIPTION
The
.Nm
utility is used to provide simplified management of
.Xr bhyve 8
virtual machines,
including networking and console access.
.Pp
Networking is handled by creating one or more virtual switches. Each switch
has a simple name which is referenced in the virtual machine configuration file.
The
.Nm
utility automatically creates a
.Xr bridge 4
device for each virtual switch and assigns virtual machine
.Xr tap 4
interfaces dynamically.
.Pp
All configuration for virtual machines is stored in a simple rc style configuration
file. When virtual machines are first created, the configuration file is copied from
a template which can be specified by the user. Multiple templates can be created providing
an easy way to provision guests with specific configurations.
.Pp
.Nm
gracefully handles reboot and shutdown commands from inside the guests, whilst providing
full management of the virtual machine from the host system.
.\" ============ BASIC SETUP ============
.Sh BASIC SETUP
Once
.Nm
is installed, create the directory which will store your virtual machine configuration and data.
This directory will be referred to as
.Pa $vm_dir
throughput this man page.
.Pp
Add the following into
.Pa /etc/rc.conf
.Bd -literal -offset indent
vm_enable="YES"
vm_dir="/your/vm/path"
vm_list=""
vm_delay="5"
.Ed
.Pp
The first and second lines are required to enable the
.Nm
utility. Please see the
.Cm startall
command description for more information on the third and fourth settings.
.Pp
Now run the
.Nm vm
.Cm init
command to finish initialisation. This will create subdirectories inside
.Pa $vm_dir
to hold configuration and templates. It will also load any required kernel modules.
This command needs to be run on each boot, which is normally handled by the rc.d script.
.Pp
Sample templates are installed to
.Pa /usr/local/share/examples/vm-bhyve/ .
You can make use of these by copying them into your
.Pa $vm_dir/.templates/
directory.
You can create and edit the templates as required. It is recommended to keep a template called
.Pa default.conf ,
as this will be used when no template is manually specified.
.\" ============ ZFS =============
.Sh ZFS
If you are using a ZFS dataset to store your virtual machines, and want a new child dataset created
for each one, specify the dataset to use in
.Pa /etc/rc.conf
as follows:
.Bd -literal -offset indent
$vm_dir="zfs:pool/dataset"
.Ed
.Pp
In contrast to earlier versions, if
.Pa $vm_dir
is a normal path, a standard subdirectory will be created for each virtual machine, regardless
of the file system type. However,
.Nm
is now able to handle situations where the dataset mountpoint does not match the dataset name.
.\" ============ QUICKSTART =============
.Sh QUICKSTART
Create a virtual switch called
.Sy public
(which is the switch name specified in the default templates) and attach it to a real interface.
Use your own interface in place of
.Sy em0
as required.
.Bd -literal -offset ident
# vm switch create public
# vm switch add public em0
.Ed
.Pp
Download an ISO file to use for installation:
.Bd -literal -offset ident
# vm iso ftp://ftp.freebsd.org/pub/FreeBSD/releases/ISO-IMAGES/10.1/FreeBSD-10.1-RELEASE-amd64-disc1.iso
.Ed
.Pp
Create a new guest using the default template and disk size, then start the installation. The
.Ar install
subcommand will immediately return you to your shell. Once started, use the
.Ar console
command to connect to the guest and complete the installation.
.Bd -literal -offset ident
# vm create my-guest
# vm install my-guest FreeBSD-10.1-RELEASE-amd64-disc1.iso
# vm console my-guest
.Ed
.Pp
Please note that Linux guests currently require the
.Sy sysutils/grub2-bhyve
package to be installed. This is used in place of
.Xr bhyveload 8
to load the guest kernel into memory.
.\" ============== WINDOWS ===============
.Sh WINDOWS SUPPORT
Windows guests are supported on versions of FreeBSD that have UEFI
support in
.Xr bhyve 8 .
As of April 2016, UEFI support should be available in FreeBSD 10.3+ and
11-CURRENT.
.Pp
You will also need a copy of the UEFI firmware. This can either be installed
using the
.Pa sysutils/uefi-edk2-bhyve
port, or you can manually download a copy (see URL below) to
.Pa $vm_dir/.config/BHYVE_UEFI.fd .
.Pp
If you are running FreeBSD 10, there is no VGA console in
.Xr bhyve 8 ,
and so an unattended installation ISO is required which allows Windows to install and
boot without any user interaction. Instructions for creating a suitable ISO can
be found at the URL below.
.Pp
.Pp
On FreeBSD 11, VGA access can be enabled by setting the
.Sy graphics="yes"
option in the guest configuration file. Once the guest has started, vnc IP & port details
can be seen in
.Sy vm list
output. See the configuration format documentation below for more detailed information on configuring graphics.
If network drivers are required, I recommend re-running the
.Sy vm install
command once the guest has been installed, but providing an ISO of the virtio-net drivers instead.
.Pp
Once the installation ISO is ready, has been placed in the
.Pa $vm_dir/.iso
directory, and you have the UEFI firmware, installation can be performed as normal.
.Bd -literal -offset indent
# vm create -t windows -s 30G winguest
# vm install winguest win_repack.iso
.Ed
.Pp
Windows installation has been tested with 2012r2 and takes around 20-25 minutes.
During install, the guest will reboot twice (three runs in total). You can see
the guest reboot by watching the log file
.Pa $vm_dir/guestname/vm-bhyve.log .
The third run should boot fully into Windows. The
.Sy virtio
network adapter will request an IP address using DHCP. Connect to the guest console
and press
.Sy i
to see the IP address that has been assigned. The default unattended installation files
should make RDP available, using Administrator and Test123 as the default login details.
.Pp
A pre-compiled copy of the UEFI firmware (BHYVE_UEFI_20160526.fd), as well as instructions for creating an
unattended installation ISO can currently be obtained from
.Pa https://people.freebsd.org/~grehan/bhyve_uefi/
.\" ============ GLOBAL OPTIONS =============
.Sh GLOBAL OPTIONS
There are some options that can be specified after
.Sy vm ,
but before any subcommand. These are global options that affect the way
.Nm
functions.
.Bl -tag -width 12n
.It Fl f
Run
.Nm
in the foreground. This option is primarily useful with the
.Nm Cm start
or
.Nm Cm install
command and runs the guest on stdio.
.It Fl i
Run the guest in interactive mode. This mode is only supported when using
the
.Sy tmux
console setting. This starts the guest on a tmux session and then immediately
connects to that session. You can detach the session, or shut the guest down
to return to your original terminal.
.El
.\" ============ SUBCOMMANDS =============
.Sh SUBCOMMANDS
.Bl -tag -width indent
.It Cm version
Show the version number of vm-bhyve installed.
.It Cm init
.br
This should be run once after each host reboot before running any other
.Nm
commands. The main function of the
.Cm init
command is as follows:
.Pp
o Load all necessary kernel modules if not already loaded
.br
o Set tap devices to come up automatically when opened
.br
o Create any configured virtual switches
.It Cm get Ar all|setting
Get a global configuration setting. These are settings that affect the functionality
of vm-bhyve, such as configuring the type of serial console to use. The keyword
.Sy all
can be used to retrieve all user configurable settings, or you can specify one or
more settings by name, separated by a space.
.It Cm set Ar setting=value
Sets the value of a global configuration setting. Multiple settings can be changed
at the same time by seperating the
.Sy setting=value
pairs with a space.
.Pp
These settings are stored in
.Pa $vm_dir/.config/system.conf
.It Cm switch list
List virtual switches. This reads all configured virtual switches from the
.Pa $vm_dir/.config/switch
file and displays them. If the virtual switches are loaded, it also tries
to display the
.Xr bridge 4
interface that has been assigned to each one.
.It Cm switch info Op Ar name Op Ar ...
This command shows detailed information about the specified virtual switch(es).
If no switch names are provided, information is output for all configured switches.
Information displayed includes the following:
.Pp
o Basic switch settings
.br
o Overall bytes sent and received via this switch
.br
o Physical ports connected
.br
o Virtual ports, including the associated virtual machine
.br
.It Xo
.Cm switch create
.Op Fl t Ar type
.Op Fl i Ar interface
.Op Fl n Ar vlan-id
.Op Fl b Ar bridge
.Op Fl m Ar mtu
.Op Fl a Ar address
.Op Fl p
.Ar name
.Xc
Create a new virtual switch. The name must be supplied and may only contain
letters, numbers and dashes. However, it may not contain a dash at the beginning
or end.
.Pp
There are currently 4 types of virtual switch that can be created. These are
.Sy standard ,
.Sy manual ,
.Sy vale
and
.Sy vxlan .
The default type is
.Sy standard ,
which creates a basic
.Xr bridge 4
interface and bridges clients to it.
.Sy manual
allows you to attach guests to a bridge that you have created and configured manually.
.Sy vale
switches use the netmap VALE system to create a virtual switch connecting guests.
.Sy vxlan
allows you to create virtual LANs (similar to a VLAN) which tunnel L2 guest traffic
over L3.
.Bl -tag -width 12n
.It Fl t Ar type
The type of virtual switch to create. The available types are listed above. This defaults
to
.Sy standard
if not specified.
.It Fl i Ar interface
For
.Sy standard
and
.Sy vxlan
switches you can attach a physical interface at creation time. This option is required
for vxlan switches.
.It Fl n Ar vlan-id
Allows you to specify a VLAN ID for
.Sy standard
and
.Sy vxlan
switches. This option is required for vxlan switches.
.It Fl b Ar bridge
If creating a manual switch using an existing bridge on your system, this option allows
you to specify the name of the bridge interface you would like to use. This option is
required for manual switches.
.It Fl m Ar mtu
Specify an mtu to use for the bridge interface.
.It Fl a Ar address
This allows you to specify an IP address that is assigned to the bridge interface.
This should be specified in
.Sy a.b.c.d/prefix-len
CIDR notation.
.It Fl p
Use this option to create a private switch. If this is enabled, no traffic will
be allowed between guests on the same switch, however then will all be able to
communicate with any physical interfaces added to the switch.
.El
.It Cm switch vlan Ar name Ar vlan-id
Assign a VLAN number to a virtual switch. The VLAN number must be between 0-4094.
.Pp
When adding an interface to a VLAN enabled virtual switch, a new
.Xr vlan 4
interface is created. This interface has the relevant parent interface and VLAN tag
configured. This vlan interface is then added to the virtual switch. As such, all
traffic between guests on the same switch is untagged and travels freely. However,
all traffic exiting via physical interfaces is tagged.
.Pp
If the virtual switch already has physical interfaces assigned, they are all removed
from the bridge, reconfigured, then re-added.
.Pp
To remove the VLAN configuration from a virtual switch, specify a
.Ar vlan-id
of 0.
.It Cm switch nat Ar name Ar on|off
Enable or disable NAT functionality on the specified switch. Please note that
.Xr pf
is required for this functionality and must be enabled in
.Pa /etc/rc.conf .
If DHCP is desired, please install the
.Xr dnsmasq
package. vm-bhyve will generate a sample dnsmasq configuration in
.Pa /usr/local/etc/dnsmasq.conf.bhyve ,
but it is up to the user to either use this configuration directly, or merge with
any existing dnsmasq settings you have configured.
.Pp
The switch should have no host ports assigned, as these will end up on the private side
of the NAT network.
.Nm
automatically detects the hosts default gateway, which is used as the forwarding interface
for NAT connections.
.Pp
Once enabled, a 172.16.X.0/24 network is assigned to the switch (bridge) interface.
.Ar X
is chosen based on the ID of the bridge interface. For example, if the switch is using
bridge10, the network will be 172.16.10.0/24.
.Xr dnsmasq
can be used to provide DHCP to the guests, and
.Xr pf
rules are inserted to provide the NAT translation.
.Pp
.Pa /etc/pf.conf
is created if it doesn't exist, and a single include statement is added. This
include statement can be moved within the file if required.
.It Cm switch private Ar on|off
Enable of disable private mode for a virtual switch. In private mode, guests will
only be able to communicate with the physical interface(s), not with each other.
.Pp
Please note that changing this setting does not affect guests that are already
running, but will be applied to any guests started from cold-boot thereafter.
.It Cm switch add Ar name Ar interface
Add the specified interface to the named virtual switch.
.Pp
The interface will immediately be added to the relevant bridge if possible, and
stored in the persistent switch configuration file. If a
.Ar vlan-id
is specified on the virtual switch, this will cause a new
.Xr vlan 4
interface to be created.
.It Cm switch remove Ar name Ar interface
Removes the specified interface from the named virtual switch and updates the
persistent configuration file.
.It Cm switch destroy Ar name
Completely remove the named virtual switch and all configuration. The associated
.Xr bridge 4
interface will be removed, as well as any
.Xr vlan 4
interfaces if they are not in use by other virtual switches.
.It Cm datastore list
List the configured datastores. Normally
.Sy vm-bhyve
will store all guests under the directory specified in
.Pa /etc/rc.conf .
This is the
.Sy default
datastore. Additional datastores can be added, providing the
ability to store guests in multiple locations on your system.
.It Cm datastore add Ar name spec
Add a new datastore to the system. The datastore name can only contain letters,
numbers and _. characters. The
.Pa spec
should use the same format as
.Sy $vm_dir .
A standard directory can be specified by just providing the path, whereas a ZFS
storage location should be specified in
.Sy zfs:pool/dataset
format.
.Pp
Please note that the directory or dataset should already exist. We do not try to
create it.
.It Cm datastore remove Ar name
Remove the specified datastore from the list. This does not destroy the directory
or dataset, leaving all files intact.
.It Cm datastore iso Ar name path
Adds a new datastore location for storing iso files. Guests cannot be created in an
iso store, but this provides an easy way to configure vm-bhyve to look in any arbitrary
location on your system (or mounted network share) where you may want to store iso images.
.It Xo
.Cm create
.Op Fl d Ar datastore
.Op Fl t Ar template
.Op Fl s Ar size
.Ar name
.Xc
Create a new virtual machine.
.Pp
Unless specified, the
.Pa default.conf
template will be used and a 20GB virtual disk image is created. This command will
created the virtual machine directory
.Pa $vm_dir/$name ,
and create the configuration file and empty disk image within.
.Bl -tag -width 12n
.It Fl d Ar datastore
Specify the datastore to create this virtual machine under. If not specified, the
.Sy default
dataset will be used, which is the location specified in
.Pa /etc/rc.conf .
.It Fl t Ar template
Specifies the template to use from within the
.Pa $vm_dir/.templates
directory. The
.Sy .conf
suffix is not required.
.It Fl s Ar size
The size of disk image to create in GB. Unless specified, the guest image will
be a sparse file 20GB in size.
.El
.It Cm destroy Ar name
Removes the specified virtual machine from the system, deleting all associated
disk images & configuration.
.It Cm list
.br
List all the virtual machines in the
.Pa $vm_dir
directory. This will show the basic configuration for each virtual machine, and whether
they are currently running.
.It Cm info Op Ar name Op Ar ...
Shows detailed information about the specified virtual machine(s). If no names are given,
information for all virtual machines is displayed.
.Pp
This output includes detailed information about network and disk devices, including
the space usage for all virtual disks (excluding custom disk devices). If the guest
is running, the output also shows the amount of host memory currently in use,
and additional network details including bytes sent/received for each virtual interface.
.It Xo
.Op Fl fi
.Cm install Ar name Ar iso
.Xc
Start a guest installation for the named virtual machine, using the specified ISO file.
The
.Ar iso
argument should be the filename of an ISO file already downloaded into the
.Pa $vm_dir/.iso
directory (or any media datastore), a full path, or a file in the current directory.
ISO files in the default .iso store can be downloaded using the
.Ar iso
subcommand described below.
.Pp
By default the installation is started in the background. Use the
.Ar console
command to connect and begin the installation.
.Pp
After installation, the guest can be rebooted and will restart using its own disk image to boot.
At this point the installation ISO file is still attached, allowing you to use the CD/DVD image
for any post installation tasks. The ISO file will remain attached after each reboot until the
guest is fully stopped.
.Pp
If the
.Ar -f
option is specified, the guest will be started in the foreground on stdio. The
.Ar -i
option starts the guest in interactive mode. This requires tmux, and the global
.Sy console
setting must be set likewise. In interactive mode the guest is started on a foreground
tmux session, but this can be detached using the standard tmux commands.
.It Xo
.Op Fl fi
.Cm start Ar name Ar ...
.Xc
Start the named virtual machine(s). The guests will boot and run completely in the background. Use
the
.Ar console
subcommand to connect to it if required.
.Pp
For each network adapter specified in the guest configuration, a
.Xr tap 4
interface will be created. If possible, the tap interface will be attached the relevant
.Xr bridge 4
interface, based on the virtual switch specified in the guest configuration.
.Pp
If the
.Ar -f
option is specified, the guest will be started in the foreground on stdio. The
.Ar -i
option starts the guest in interactive mode. This requires tmux, and the global
.Sy console
setting must be set likewise. In interactive mode the guest is started on a foreground
tmux session, but this can be detached using the standard tmux commands.
.It Cm stop Ar name Ar ...
Stop a named virtual machine. All
.Xr tap 4
and
.Xr nmdm 4
devices will be automatically cleaned up once the guest has exited.
.Pp
If a guest is stuck in the bootloader stage, you are given the option to forcibly stop it.
.Pp
Multiple guests can be specified to this command at the same time. Each one will be sent a
poweroff event.
.It Cm console Ar name Op Ar com1|com2
Connect to the console of the named virtual machine. Without network access, this is the primary
way of connecting to the guest once it is running.
.Pp
By default this will connect to the first com port specified in the client configuration, which
is usually com1. Alternatively you can specify the com port to connect to.
.Pp
This looks for the
.Xr nmdm 4
device associated with the virtual machine, and connects to it with
.Xr cu 1 .
Use ~+Ctrl-D to exit the console and return to the host.
.It Cm rename Ar name Ar new-name
Renames the specified virtual machine. The guest must be stopped to use this function.
.It Xo
.Cm add
.Op Fl d Ar device
.Op Fl t Ar type
.Op Fl s Ar size|switch
.Ar name
.Xc
Add a new network or disk device to the named virtual machine. The options depend on
the type of device that is being added:
.Bl -tag -width 15n
.It Fl d Ar device
The type of device to add. Currently this can either be
.Pa disk
or
.Pa network
.It Fl t Ar type
For disk devices, this specifies the type of disk device to create.
Valid options for this are
.Pa zvol ,
.Pa sparse-zvol
and
.Pa file .
If not specified, this defaults to
.Pa file .
.It Fl s Ar size|switch
For disk devices, this is used to specify the size of the disk image to create. For
network devices, use this option to specify the virtual switch to connect the network interface to.
.El
.Pp
For both types of device, the emulation type will be chosen automatically based on the
emulation used for the existing guest devices.
.It Cm reset Ar name
Forcefully reset the named virtual machine. This can cause corruption to the guest file system just
as with real hardware and should only be used if necessary.
.It Cm poweroff Ar name
Forcefully power off the named virtual machine. As with
.Ar reset
above, this does not inform the guest to shutdown gracefully and should only be used if the guest
can not be shut down using normal methods.
.It Cm startall
Start all virtual machines configured for auto-start. This is the command used by the rc.d scripts
to start all machines on boot.
.Pp
The list of virtual machines should be specified using the
.Pa $vm_list
variable in
.Pa /etc/rc.conf .
This allows you to use shared storage for virtual machine data, whilst making sure that the correct
guests are started automatically on each host. (Or to just make sure your required guests start on boot
whilst leaving test/un-needed guests alone)
.Pp
The delay between starting guests can be set using the
.Pa $vm_delay
variable, which defaults to 5 seconds. Too small a delay can cause problems, as each guest doesn't
have enough time to claim a null modem device before the next guest starts. Increasing this value
can be useful if you have disk-intensive guests and want to give each guest a chance to fully
boot before the next starts.
.It Cm stopall
Stop all running virtual machines. This sends a stop command to all
.Xr bhyve 8
instances, regardless of whether they were starting using
.Nm
or not.
.It Cm configure Ar name
The
.Cm configure
command simply opens the virtual machine configuration file in your default editor,
allowing you to easily make changes. Please note, changes do not take effect until
the virtual machine is fully shutdown and restarted.
.It Cm passthru
The
.Cm passthru
command lists all PCI devices in the system, the device ID required for bhyve, and
whether the device is currently ready to be used by a guest. In order to make a
device ready, it needs to be reserved on boot by adding the device ID to the
.Sy pptdevs
variable in
.Pa /boot/loader.conf .
.Pp
Once a device is ready, it can be assigned to a guest by adding
.Sy passthruX="{ID}"
to the guest's configuration file.
.Sy X
should be an integer starting at 0 for the first passthrough device.
.Pp
More details can be found in the bhyve wiki.
.It Cm clone Ar name[@snapshot] Ar new-name
Create a clone of the virtual machine
.Pa name ,
as long as it is currently powered off. The new machine will be called
.Pa new-name ,
and will be ready to boot with a newly assigned UUID and empty log file.
.Pp
If no snapshot name is given, a new snapshot will be taken of the guest and any descendant
datasets or ZVOLs. If you wish to use an existing snapshot as the source for the clone,
please make sure the snapshot exists for the guest and any child ZVOLs, otherwise the clone
will fail.
.Pp
Please note that this function requires ZFS.
.It Xo
.Cm snapshot
.Op Fl f
.Ar name|name@snapshot
.Xc
Create a snapshot of the names virtual machine. This command is only supported with ZFS
and will take a snapshot of the guest dataset and any descendant ZVOL devices.
.Pp
The guest and snapshot name can be specified in the normal
.Pa name@snapshot
way familiar to ZFS users. If no snapshot name is given, the snapshot is based on the current timestamp in
.Pa Y-m-d-H:M:S
format.
.Pp
By default the guest must be stopped to use this command, although you can force a snapshot
of a running guest by using the
.Fl f
option.
.It Xo
.Cm rollback
.Op Fl r
.Ar name@snapshot
.Xc
Rollback the guest to the specified snapshot. This will roll back the guest dataset and all
descendant ZVOL devices.
.Pp
Normally, ZFS will only allow you to roll back to the most recent snapshot.
If the snapshot given is not the most recent, ZFS will produce a warning detailing that you
need to use the
.Fl r
option to remove the more recent snapshots. It will also produce a list of the snapshots that
will be destroyed if you use this option. The
.Fl r
option can be passed directly into
.Nm
.Cm rollback
.Pp
The guest must always be stopped to use this command.
.It Xo
.Cm send
.Op Fl 12t
.Op Fl i Ar snapshot
.Ar name
.Ar host:port
.Xc
Send a guest to the specified host. The receiving end must already be in recv mode (see
.Ar recv
command below), which will provide the port number to use. This relies on ZFS features.
By default this is a two stage procedure. The first stage sends a full snapshot of the guest.
Once this is complete the guest is shutdown using an ACPI shutdown (if required) and then
an incremental snapshot is sent. If required, the guest can be automatically booted on the
receiving host once completed.
.Pp
Please note that any virtual switches used by the guest should exist on both hosts, and
if using UEFI, the firmware must be available on both. It is also advised that the receiver
is running a matching or newer version of FreeBSD.
.Pp
The function is not supported if the guest has passthrough devices or custom disks.
.Bl -tag -width 12n
.It Fl i Ar snapshot
If the receiving end already has a copy of the guest, use this to specify an existing snapshot
to use as the incremental source, rather than sending a full snapshot. A full snapshot will fail
if the guest already exists on the remote host.
.Pp
If only running stage 2, the guest must already exist on the receiving side and this option is required.
.It Fl 1
Only run the first stage. This will send a single snapshot and then exit. The receiving end
must be run in the same mode, and is useful for sending the first major snapshot without interrupting
the guest.
.It Fl 2
Only run the second stage. If the remote end already has an up-to-date snapshot, this can be used to
shut the guest down (if required), send a incremental snapshot, then boot the new guest.
.It Fl t
This runs triple stage mode, useful for large guests. The basic premise behind this is that the first
full send may take several hours, or longer, for a large guest. During this time many changes could be made. We then
run a second stage while the guest is still running to send these changes. This should hopefully send less
data, and so complete quicker. Finally we shutdown the guest and do a third send. If the second send completed
quicker than the first, there should be far fewer changes to send during this third stage, resulting in
minimal downtime.
.El
.It Xo
.Cm recv
.Op Fl s12
.Op Fl d Ar datastore
.Ar name
.Xc
Prepare to receive a guest. This outputs a port number which should be provided to the
.Ar send
command used on the sending host.
.Bl -tag -width 12n
.It Fl d Ar datastore
Create the new guest on the specific datastore instead of the
.Sy default
store.
.It Fl s
Start the guest as soon as stage 2 has been completed.
.It Fl 1
Only run the first stage. This receives a snapshot and then exits. This is useful to get the full
snapshot sent, allowing you to complete the second stage during a pre-determined maintenance window.
.It Fl 2
Only run the second stage. This expects to receive an incremental snapshot for an existing guest,
and will attempt to start the guest (if the
.Sy -s
option is also provided).
.It Fl t
Run the receiver in triple stage mode which expects three snapshots. See the
.Cm send
command above for details on this mode.
.El
.Xc
.It Cm iso Op Ar url
List all the ISO files currently stored in the
.Pa $vm_dir/.iso
directory. This is often useful during guest installation, allowing you to copy and paste the ISO
filename.
.Pp
If a
.Sy url
is specified, instead of listing ISO files, it attempts to download the given file using
.Xr fetch 1 .
.It Cm image list
List available images. Any virtual machine can be packaged into an image, which can then be
used to create additional machines. All images have a globally unique ID (UUID) which is
used to identify them. The list command shows the UUID, the original machine name, the
date it was created and a short description of the image.
.Pp
Please note that these commands rely on using ZFS featured to package/unpackage the images,
and as such are only available when using a ZFS dataset as the storage location.
.It Xo
.Cm image create
.Op Fl d Ar description
.Op Fl u
.Ar name
.Xc
Create a new image from the named virtual machine. This will create a compressed copy of
the original guest dataset, which is stored in the
.Pa $vm_dir/images
directory. It also creates a
.Pa UUID.manifest
file which contains details about the image.
.Pp
Once complete, it will display the UUID which has been assigned to this image.
.Pp
If you do not want the image to be compressed, specify the
.Sy -u
option.
.It Xo
.Cm image provision
.Op Fl d Ar datastore
.Ar uuid Ar new-name
.Xc
Create a new virtual machine, named
.Pa new-name ,
from the specified image UUID. This will be created on the
.Sy default
datastore unless specified otherwise.
.It Cm image destroy Ar uuid
Destroy the specified image.
.El
.\" ============ GLOBAL CONFIGURATION ===========
.Sh GLOBAL CONFIGURATION
These configuration options are stored in
.Pa $vm_dir/.config/system.conf ,
and affect the global functionality of vm-bhyve. These settings can be changed by
either editing the configuration file manually, or using the
.Sy vm set
and
.Sy vm get
commands.
.Bl -tag -width 17n
.It console
Set the type of console to use, which defaults to
.Sy nmdm .
If you have the tmux port installed and would prefer to use that for guest
console access, you can set this option to
.Sy tmux .
.El
.\" ============ CONFIGURATION FORMAT ===========
.Sh GUEST CONFIGURATION FORMAT
Each virtual machine has a configuration file that specifies the hardware configuration. This
uses a similar format to the
.Sy rc
files, making them easy to edit by hand. The settings for each guest are stored in
.Pa $vm_dir/$vm_name/$vm_name.conf .
An overview of the available configuration options is listed below.
.Bl -tag -width 17n
.It loader
Windows, Linux & FreeBSD guests will use the correct loader by default. For other
guests that require a loader to be used, this can set to
.Sy bhyveload