fides Quick Start Troubleshooting #1407

daveqnet-alt · 2022-10-12T13:55:47Z

Bug Description

Hi fides team,

I'm trying to get fides up and running locally for the very first time but am having some problems. I hope you can help. Basically:

I get WARN messages about variables not being set during quick start. Can I ignore these safely?
Why does the first quick start build take ~ 10 minutes (592 seconds in installation log attached below)? This is not quick. Is there some way of speeding this up?
The quick start guide says to wait until the fides# prompt appears. It never appears.
After nox -s dev is finished, how do I exit?

Steps to Reproduce

Clean your Docker env to simulate a first time user's experience
1. Stop all running containers docker stop $(docker ps -a -q)
2. Remove all containers docker rm $(docker ps -a -q)
3. Prune all images docker image prune -a
4. Prune all volumes docker volume prune
5. Prune build cache docker builder prune -a.
Folllow fides quick start instructions here
Try to reach the fides init part of Step 2 (I didn't get past nox -s dev).

Expected behavior

No WARN messages
First-time quick start of ~ 3 minutes (this is the expected time stated in the docs)
A fides# prompt (maybe I'm supposed to docker exec into a running container or something?)
Guidance on how to exit / tear down quick start env.

Screenshots

I'm attaching the entire installation log here:

2022-10-12 daveqnet-alt fides quick start installation.log

The WARN messages were...

WARN[0000] The "VAULT_ADDR" variable is not set. Defaulting to a blank string.                                                                                                                                                                                                            
WARN[0000] The "VAULT_TOKEN" variable is not set. Defaulting to a blank string. 
WARN[0000] The "FIDES__CLI__ANALYTICS_ID" variable is not set. Defaulting to a blank string. 
WARN[0000] The "VAULT_NAMESPACE" variable is not set. Defaulting to a blank string. 
WARN[0000] The "FIDES__CLI__ANALYTICS_ID" variable is not set. Defaulting to a blank string.

Environment

Version: v1.9.2, I think? Not sure given recent repo unification efforts. See below for exact commit to main
OS: See below
Python Version: See below
Docker Version: See below

~/projects/github/ethyca/fides main ❯ git log -1 | head -n 3
commit eb958dad3e313b6787199c4454c9493ac3595406
Author: Adam Sachs <[email protected]>
Date:   Tue Oct 11 17:02:09 2022 -0400
~/projects/github/ethyca/fides main ❯ system_profiler SPSoftwareDataType SPHardwareDataType
Software:

    System Software Overview:

      System Version: macOS 12.6 (21G115)
      Kernel Version: Darwin 21.6.0
      <redacted>

Hardware:

    Hardware Overview:

      Model Name: MacBook Pro
      Model Identifier: MacBookPro17,1
      Chip: Apple M1
      Total Number of Cores: 8 (4 performance and 4 efficiency)
      Memory: 16 GB
      System Firmware Version: 7459.141.1
      OS Loader Version: 7459.141.1
      <redacted>

~/projects/github/ethyca/fides main ❯ python --version
Python 3.10.7
~/projects/github/ethyca/fides main ❯ nox --version
2022.8.7
~/projects/github/ethyca/fides main ❯ pipx --version
1.1.0
~/projects/github/ethyca/fides main ❯ docker --version
Docker version 20.10.17, build 100c701
~/projects/github/ethyca/fides main ❯ docker-compose --version
Docker Compose version v2.10.2

Additional context

This is obviously an issue raised by an alt of @daveqnet, but please treat it exactly the same as you would an issue raised by any non-Ethyca GitHub member

The text was updated successfully, but these errors were encountered:

sanders41 · 2022-10-12T14:41:01Z

I get WARN messages about variables not being set during quick start. Can I ignore these safely?

Yes, you can safely ignore these. They are more info than warning, we might should change it to info.

Why does the first quick start build take ~ 10 minutes (592 seconds in installation log attached below)? This is not quick. Is there some way of speeding this up?

This is an issue with M1/M2 macs that was just fixed. If you pull the latest main you will get this update.

The quick start guide says to wait until the fides# prompt appears. It never appears.

The quick start command is missing an argument. It should be nox -s dev -- shell. I have opened an issue for this here. This is the biggest part of your issue with completing the quick start. If you have any more issues after nox -s dev -- shell feel free to follow-up with more questions.

After nox -s dev is finished, how do I exit?

Control + c will stop the container.

daveqnet-alt · 2022-10-12T15:07:54Z

Thank you for your help on this, @sanders41!

After a git pull the build time in a cleaned Docker env (no cache) shrank to 158 seconds, which is a HUGE improvement. 🎉

Using nox -s dev -- shell as you suggested also takes me to a fides# prompt. 🥳

Just on the env teardown, ctrl+c in the container itself doesn't work...

+] Running 3/0
 ⠿ Container fides-fides-db-1  Running                                                                                                                                                                                                                                               0.0s
 ⠿ Container fides-redis-1     Running                                                                                                                                                                                                                                               0.0s
 ⠿ Container fides-worker-1    Running                                                                                                                                                                                                                                               0.0s
root@3ddb6ea288f3:/fides# ^C
root@3ddb6ea288f3:/fides#

but if I exit the container and then use ctrl+c it seems to work.

root@3ddb6ea288f3:/fides# exit
exit
nox > Command docker compose run --rm -e FIDES__CLI__ANALYTICS_ID -e ANALYTICS_OPT_OUT --user=root fides /bin/bash failed with exit code 130
nox > Session dev failed.
nox > Running session teardown
nox > Creating virtual environment (virtualenv) using python3.10 in .nox/teardown
nox > docker compose -f docker-compose.yml -f docker-compose.integration-tests.yml -f docker/docker-compose.integration-mariadb.yml -f docker/docker-compose.integration-mongodb.yml -f docker/docker-compose.integration-mysql.yml -f docker/docker-compose.integration-postgres.yml -f docker/docker-compose.integration-mssql.yml down --remove-orphans
WARN[0000] The "FIDES__CLI__ANALYTICS_ID" variable is not set. Defaulting to a blank string. 
WARN[0000] The "VAULT_ADDR" variable is not set. Defaulting to a blank string. 
WARN[0000] The "FIDES__CLI__ANALYTICS_ID" variable is not set. Defaulting to a blank string. 
WARN[0000] The "VAULT_NAMESPACE" variable is not set. Defaulting to a blank string. 
WARN[0000] The "VAULT_TOKEN" variable is not set. Defaulting to a blank string. 
[+] Running 5/5
 ⠿ Container fides-fides-1     Removed                                                                                                                                                                                                                                               1.6s
 ⠿ Container fides-fides-db-1  Removed                                                                                                                                                                                                                                               0.2s
 ⠿ Container fides-worker-1    Removed                                                                                                                                                                                                                                               2.2s
 ⠿ Container fides-redis-1     Removed                                                                                                                                                                                                                                               0.2s
 ⠿ Network fides_default       Removed                                                                                                                                                                                                                                               0.1s
Teardown complete
nox > Session teardown was successful.
nox > Ran multiple sessions:
nox > * dev: failed
nox > * teardown: success

Should I be concerned about the nox > * dev: failed message?

Also, the volumes created during the quick start persist - is this intended?

~/projects/github/ethyca/fides main ❯ docker ps -a                                                                                                                                                                                                                                 2m 58s
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
~/projects/github/ethyca/fides main ❯ docker image ls
REPOSITORY     TAG            IMAGE ID       CREATED         SIZE
ethyca/fides   local          752a2c5ef451   3 minutes ago   1.27GB
<none>         <none>         3fc3af312eca   3 minutes ago   1.27GB
postgres       12             bfb204038175   7 days ago      352MB
redis          6.2.5-alpine   2d113b08aca4   13 months ago   32.4MB
~/projects/github/ethyca/fides main ❯ docker volume ls  
DRIVER    VOLUME NAME
local     1ee8875cf7fe78fc454fa391878e39868642a206ef5289d26ffb6cc833b50822
local     ff258eb3117718b400dab2d36c09279b47cde34fdea14c849afc4666f0564f04
local     fides_postgres

sanders41 · 2022-10-12T15:17:49Z

Sorry, I should have been more specific on the stopping. ctrl + c is how to stop after running nox -s dev. As you found, exit is what you need when using nox -s dev -- shell, you won't even need the ctrl + c with this one, exit will be enough.

Should I be concerned about the nox > * dev: failed message?

Occasionally there is an issue with stopping containers when exiting, nothing to be concerned about. You can check if this is the case by running docker ps. If you still see any containers running you can stop them with docker stop <abc123> where <abc123> is the CONTAINER ID that shows after running the docker ps command.

Also, the volumes created during the quick start persist - is this intended?

This is expected. If you want to remove the volumes you can run nox -s teardown -- volumes

daveqnet-alt · 2022-10-12T23:13:47Z

Thanks again, @sanders41, I'm a little further along in the quick start instructions. At Step 3 though I encountered a new error and the generated demo datamap .xlsx (attached below) is blank.

Any idea what's going wrong here?

root@3df826f9ff68:/fides# fides init
Loading config from: /fides/.fides/fides.toml

███████╗██╗██████╗ ███████╗███████╗
██╔════╝██║██╔══██╗██╔════╝██╔════╝
█████╗  ██║██║  ██║█████╗  ███████╗
██╔══╝  ██║██║  ██║██╔══╝  ╚════██║
██║     ██║██████╔╝███████╗███████║
╚═╝     ╚═╝╚═════╝ ╚══════╝╚══════╝

Initializing fides...
----------
Directory './.fides' already exists.
----------
Configuration file already exists: ./.fides/fides.toml
----------
For example policies and help getting started, see:
	https://ethyca.github.io/fides/guides/policies/
----------
fides initialization complete.
root@3df826f9ff68:/fides# fides push demo_resources/
Loading config from: /fides/.fides/fides.toml
Loading resource manifests from: demo_resources/
Taxonomy successfully created.
----------
Processing organization resource(s)...
{'detail': {'error': 'a database query failed'}}
root@3df826f9ff68:/fides# fides export datamap --output-dir demo_resources/
Loading config from: /fides/.fides/fides.toml
2022-10-12-T230322_datamap.xlsx successfully exported.

2022-10-12-T230322_datamap.xlsx

daveqnet-alt · 2022-10-12T23:37:15Z

Okay, I'm not sure why it didn't work the first time, but just trying it a second time seemed to work.

root@3df826f9ff68:/fides# rm ./demo_resources/2022-10-12-T230322_datamap.xlsx 
root@3df826f9ff68:/fides# fides push demo_resources/
Loading config from: /fides/.fides/fides.toml
Loading resource manifests from: demo_resources/
Taxonomy successfully created.
----------
Processing policy resource(s)...
PUSHED 1 policy resource(s).
----------
Processing dataset resource(s)...
PUSHED 1 dataset resource(s).
----------
Processing data_subject resource(s)...
PUSHED 1 data_subject resource(s).
----------
Processing registry resource(s)...
PUSHED 1 registry resource(s).
----------
Processing system resource(s)...
PUSHED 2 system resource(s).
----------
Processing data_use resource(s)...
PUSHED 1 data_use resource(s).
----------
Processing organization resource(s)...
PUSHED 1 organization resource(s).
----------
root@3df826f9ff68:/fides# fides export datamap --output-dir demo_resources/
Loading config from: /fides/.fides/fides.toml
2022-10-12-T233237_datamap.xlsx successfully exported.

2022-10-12-T233237_datamap.xlsx

I will continue...

daveqnet-alt · 2022-10-12T23:44:17Z

Unfortunately I'm stuck on Step 5 now. fidesctl doesn't seem to exist in this container?

root@3df826f9ff68:/fides# fidesctl evaluate demo_resources/ --audit
bash: fidesctl: command not found
root@3df826f9ff68:/fides# which fidesctl
root@3df826f9ff68:/fides# whereis fidesctl
fidesctl:
root@3df826f9ff68:/fides# find /  -name "fidesctl"
root@3df826f9ff68:/fides# echo $PATH
/root/.local/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

daveqnet-alt · 2022-10-12T23:59:43Z

Figured it out: there's an error in the quick start guide, the command should be fides evaluate demo_resources/ --audit, not fidesctl evaluate demo_resources/ --audit

root@3df826f9ff68:/fides# fides evaluate demo_resources/ --audit
Loading config from: /fides/.fides/fides.toml
Loading resource manifests from: demo_resources/
Taxonomy successfully created.
----------
Processing system resource(s)...
PUSHED 2 system resource(s).
----------
Processing data_subject resource(s)...
PUSHED 1 data_subject resource(s).
----------
Processing data_use resource(s)...
PUSHED 1 data_use resource(s).
----------
Processing organization resource(s)...
PUSHED 1 organization resource(s).
----------
Processing policy resource(s)...
PUSHED 1 policy resource(s).
----------
Processing registry resource(s)...
PUSHED 1 registry resource(s).
----------
Processing dataset resource(s)...
PUSHED 1 dataset resource(s).
----------
Loading resource manifests from: demo_resources/
Taxonomy successfully created.
Evaluating the following policies:
- demo_privacy_policy
----------
Checking for missing resources...
Executing Policy evaluation(s)...
Sending the evaluation results to the server...
Evaluation passed!
----------
'Auditing Organization Resource Compliance'
Found 1 Organization resource(s) to audit...
Auditing Organization: Demo Organization
controller for default_organization in Demo Organization is compliant
data_protection_officer for default_organization in Demo Organization is compliant
representative for default_organization in Demo Organization is compliant
security_policy for default_organization in Demo Organization is compliant
All audited organization resource(s) compliant!
----------
'Auditing System Resource Compliance'
Found 2 System resource(s) to audit...
'Auditing System: Demo Analytics System'
improve.system missing recipients in Demo Analytics System.
improve.system missing legal_basis in Demo Analytics System.
improve.system missing special_category in Demo Analytics System.
customer missing rights in Demo Analytics System.
customer missing automated_decisions_or_profiling in Demo Analytics System.
'Auditing System: Demo Marketing System'
advertising missing recipients in Demo Marketing System.
advertising missing legal_basis in Demo Marketing System.
advertising missing special_category in Demo Marketing System.
customer missing rights in Demo Marketing System.
customer missing automated_decisions_or_profiling in Demo Marketing System.
10 issue(s) were detected in auditing system completeness.

ThomasLaPiana · 2022-10-13T03:23:00Z

@daveqnet-alt thanks for running through this Dave! The README is sorely out-of-date, and something we need to fix ASAP

daveqnet-alt · 2022-10-13T07:41:07Z

Thanks @ThomasLaPiana, and fair enough, I understand the documentation needs to be updated.

However, is it worth taking a step up and looking at the overall journey for a novice external user visiting the newly unified fides repo for the first time? This is what I was trying to simulate for you and the team in this issue and, although this is a sample of n = 1, it was not smooth sailing.

For a quick start in unified fides, what will be of most value to a novice to get up and running in a few minutes and entice them to explore further? Maybe exposure to a UI (I'm pretty sure there are multiple UIs now)? It's also not clear to me where what used to be fidesops fits into a quick start. Questions for Product internally more than yourself maybe.

Anyway, to list out the bumps I experienced, if it helps:

When I tried to run nox -s dev for the first time over the weekend it resulted in a bunch of errors like ERROR:uvicorn.error:Application startup failed. Exiting., which I presumed to mean that Step 1 of the quick start had failed. I didn't mention it in this issue before, since the failures disappeared when I tried it a few days and commits to main later. Maybe the fix here is to guide a user during quick start to to work from a release branch, not main, as main is not stable? Anyway, I've attached the installation failure log at the bottom of this comment if you're interested.
nox -s dev -- shell should have been the initial command. Simple fix: Fix shell command in README quickstart #1412
WARN log messages for missing variables in the quick start. Consider dropping to INFO, as Paul suggested.
Extremely long build times on an Apple Silicon Mac. Fair play to you, this has already been fixed. I want to repeat my earlier comment: going from a 10 min to a 3 min build is a HUGE improvement. 👏
My first attempt to follow Step 3 and export a datamap resulted in a blank datamap (due to a database error?). A second attempt worked, but this seems flaky. It should work first time in a quick start / demo.
fidesctl not working in Step 5. I guess (but am not sure) that it has been replaced by fides everywhere, so this could just be a global find and replace in documentation?
This is a very simple suggestion, but a final step in the quick start guide explaining how to pause or tear down the fides environment generated during the quick start would be useful to a novice (see my back and forth about ctrl+c and exit with Paul in a comment earlier above).

Finally, @sanders41, thank you again for your patience with me and for walking me through the issues until I got the quick start env working. 🙏

2022-10-09 Dave Q fides nox -s dev installation failure.log

ThomasLaPiana · 2022-10-13T08:00:36Z

@daveqnet-alt I (and the whole engineering team!) hugely appreciate you taking the time to go through the README and find all of these bugs/problem areas. It is really valuable to us, and I agree that sometimes we're so in the weeds we forget to focus on the foundational basics.

Paul and I will iron out the README and maybe you'd be happy to give it another go once we do?

daveqnet-alt · 2022-10-13T08:02:01Z

Paul and I will iron out the README and maybe you'd be happy to give it another go once we do?

Perfect, let's do that, thanks! 😁

daveqnet · 2023-01-10T13:46:25Z

I'm closing this issue as the changes made in fides 2.x e.g. fides deploy up obsoleted the fides 1.x quickstart guide.

daveqnet-alt added the bug Something isn't working label Oct 12, 2022

This was referenced Oct 13, 2022

Fides push fails randomly with database query failed error #1435

Closed

Some warnings at startup should instead be info #1434

Closed

daveqnet closed this as completed Jan 10, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fides Quick Start Troubleshooting #1407

fides Quick Start Troubleshooting #1407

daveqnet-alt commented Oct 12, 2022 •

edited by mfbrown

Loading

sanders41 commented Oct 12, 2022 •

edited

Loading

daveqnet-alt commented Oct 12, 2022

sanders41 commented Oct 12, 2022

daveqnet-alt commented Oct 12, 2022

daveqnet-alt commented Oct 12, 2022 •

edited

Loading

daveqnet-alt commented Oct 12, 2022

daveqnet-alt commented Oct 12, 2022

ThomasLaPiana commented Oct 13, 2022

daveqnet-alt commented Oct 13, 2022 •

edited

Loading

ThomasLaPiana commented Oct 13, 2022

daveqnet-alt commented Oct 13, 2022

daveqnet commented Jan 10, 2023

fides Quick Start Troubleshooting #1407

fides Quick Start Troubleshooting #1407

Comments

daveqnet-alt commented Oct 12, 2022 • edited by mfbrown Loading

Bug Description

Steps to Reproduce

Expected behavior

Screenshots

Environment

Additional context

sanders41 commented Oct 12, 2022 • edited Loading

daveqnet-alt commented Oct 12, 2022

sanders41 commented Oct 12, 2022

daveqnet-alt commented Oct 12, 2022

daveqnet-alt commented Oct 12, 2022 • edited Loading

daveqnet-alt commented Oct 12, 2022

daveqnet-alt commented Oct 12, 2022

ThomasLaPiana commented Oct 13, 2022

daveqnet-alt commented Oct 13, 2022 • edited Loading

ThomasLaPiana commented Oct 13, 2022

daveqnet-alt commented Oct 13, 2022

daveqnet commented Jan 10, 2023

daveqnet-alt commented Oct 12, 2022 •

edited by mfbrown

Loading

sanders41 commented Oct 12, 2022 •

edited

Loading

daveqnet-alt commented Oct 12, 2022 •

edited

Loading

daveqnet-alt commented Oct 13, 2022 •

edited

Loading