Having trouble loading assets in a notebook

[seeess1]

Hi! I tackled my first PUDL ticket here (relevant issue here). For the most part things went pretty smooth. But I was never able to actually load the asset locally into a notebook with:

from dagster import AssetKey
from pudl.etl import defs

demand_forecasts = defs.load_asset_value(AssetKey(<name of asset>))

So I had to do a sorta clunky workaround (you can see what I mean in notebooks/work-in-progress/ferc714-core.ipynb in my merged PR). Doing that unblocked me but I came back afterward to try and figure out how to load assets using the logic posted above. I ended up trying a few things including rebuilding my pudl environment from scratch (which had its own complications) but nothing worked. Am hoping to get some advice/potential solutions here on how I can set up my machine so I can easily load assets into notebooks/scripts for local development.

Also, for context, I have all the raw/SQLite PUDL data from the ETL in a directory called pudl-data that sits next to the pudl repo on my machine. So my dagster_home lives here:

/Users/sam/Documents/pudl-data/dagster_home

And pudl_output lives here:

/Users/sam/Documents/pudl-data/pudl_output

And my pudl repo lives here:

/Users/sam/Documents/pudl

[zaneselvans]

Okay, some questions to debug with...

• What operating system are you working in?
• In your newly recreated pudl-dev environment are you able to run dagster at the commnd line, and materialize assets using the Dagster UI / web interface?
• Have you already materialized the asset (through the Dagster UI) that you're trying to load?
• Have you tried running that very short 3-line script above to load an asset from within a python prompt in the terminal with the pudl-dev environment activated?
• Can you past the error messages that you're getting when you try to run the above script in your notebook?
• How are you running the Jupyter Notebook in which you're unable to load the asset? Are you sure it's running within the pudl-dev environment? How can you check?
• Can you read your $DAGSTER_HOME and $PUDL_OUTPUT environment variables from within the notebook? For example:

import os
print(os.environ["DAGSTER_HOME"])
# '/Users/zane/code/catalyst/dagster-home'
print(os.environ["PUDL_OUTPUT"])
# '/Users/zane/code/catalyst/pudl-output'

Similarly, from within the notebook are you able to import libraries that should be available in pudl-dev?

import dagster
import pudl

[zaneselvans]

I suspect it's easiest to just download the DB from the nightly builds, which should match the alembic migrations on main and create then a new feature branch.

If you're just working on one branch where you need to change the schema, and the schema isn't changing on main at the same time this is pretty simple. As you add or change tables, you'll create migrations on your branch, and when you merge the branch into main they'll go along for the ride. It can also be nice to squash all of the accumulated DB migrations from your branch into a single migration before merging into main by downgrading to the last shared revision, removing the incremental migrations, and then creating a single new migration that contains all of the accumulated changes. If you're switching between different branches that have different schema changes happening, or there are changes on main that you need to bring into your own branch it gets more complicated.

[seeess1]

Assets/Dagster

After running the aws and unzip commands, I'm able to materialize the core_ferc714__yearly_planning_area_demand_forecast asset now, which is progress! But I also tried materializing another one (_core_censusdp1tract__counties) that currently shows up as failed when I load Dagster and it errored out with sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) unable to open database file.

Environment/shell

On the environment/shell front, I did some more testing there. When I run a terminal in the VS Code GUI, it defaults to bash:

But notebooks are still failing to load assets as you can see in the screenshot above.

So I tried using zsh instead. But when I go to activate the environment (mamba activate pudl-dev), I get:

Run 'mamba init' to be able to run mamba activate/deactivate and start a new shell session. Or use conda to activate/deactivate.

Once I run mamba init, I get a list of files with no change and one line saying that /Users/sam/.bash_profile has been modified. At the end it says:

For changes to take effect, close and re-open your current shell.

But if I close the shell, open a new terminal, and then run mamba activate pudl-dev, I go right back to Run 'mamba init' to be able to... 😵‍💫.

At this point, I'm honestly okay with not being able to run notebooks. It would be nice if I could but I can load assets in regular Python scripts, which should be good enough for me. If I can just resolve the asset materialization issue that I noted at the top that should unblock me. Any thoughts on that one?

[zaneselvans]

Can you post the entire SQLite database error? There are several databases potentially in play here -- the Dagster has one that it uses to log events, there's the SQLite DB, and then there's a separate Census DP1 database, and I'm not sure which one the error is related to.

Do you have censusdp1tract.sqlite in your $PUDL_OUTPUT directory?

With the notebook, I think it'll be challenging to explore and debug a new asset without some way to interactively play with the outputs.

What is your login shell? If you open a new terminal (not in VS Code) and run echo $SHELL what do you get? If you open a new terminal in VS Code, do you get the same thing?

[zaneselvans]

Rather than materializing a single asset, you might also try using *censusdp1 in the search bar of the asset graph to select all assets upstream of those that contain the string censusdp1 and then doing "Materialize All" -- if the problem is that you don't have the Census DP1 database locally that should regenerate it.

The docs on Dagster's asset selection syntax are really helpful.

[seeess1]

I'm good now on _core_censusdp1tract__counties! Looked up the related assets like you recommended and materialized everything in the lineage (just a group of four assets) and it all went through. (The original error logs are attached. And before I ran the materialization I checked and confirmed that I had the censusdp1tract.sqlite file in my $PUDL_OUTPUT directory.)

Yeah I hear you on the complication with not being able to use a notebook. When I run echo $SHELL both in a regular terminal and in a a VS Code terminal, I get /bin/bash.