Add full ASTM G-173-03 tables

[echedey-ls]

Is your feature request related to a problem? Please describe.
Add a way of reading the extraterrestrial and direct components from ASTM G-173-03 tables, provided by NREL.

Describe the solution you'd like
Add two other functions like get_am15g, e.g. get_am15b and get_am15extra, to replicate the same and read those other two columns.

Describe alternatives you've considered
Add this feature into get_am15g (and possibly rename it) to avoid replicating code.

Additional context
Current global spectra file is 29.1kB, the original CSV provided by NREL is 58.3kB.

I need to read the direct spectra for a small research. Just in case you think it's a good addition, I can add functions for all of that.

[mikofski]

It would also be useful to have access to quantum efficiency of several popular cell tech

[echedey-ls]

It would also be useful to have access to quantum efficiency of several popular cell tech

@mikofski, I've been wandering what EQE stands for lately. That would be great of course! If it's an easy task, or better, requires the inclusion of scientific models I can add it to my final-year thesis TODO list 😉

[mikofski]

Talk to @shirubana NREL has several measured QE curves. I don’t know about scientific models, but with solar spectrum, am1.5, and QE you can calculate the spectral correction: https://gist.github.com/mikofski/54aac3ee295bab8118df

[markcampanelli]

PVfit uses custom classes to validate and store spectra and QEs as “data functions”:

https://github.com/markcampanelli/pvfit/blob/main/pvfit/measurement/spectral_correction/types.py

The demos have a example of loading a standard spectrum from ASTM G-173:

https://github.com/markcampanelli/pvfit/blob/main/pvfit/measurement/spectral_correction/demos/data.py

Part of the design philosophy for using the data-function classes was to move the validation into the class construction so that the functions subsequently operating on this data can cleanly focus on the algorithm (e.g., spectral correction M computations). There are also convenience methods like conversion from QE to SR. I prefer to keep the algorithms as pure function on data (classes), in more of a functional programming style.

[adriesse]

PV module spectral response measurements - Data and Resources

https://www.osti.gov/biblio/2204677

[mikofski]

@adriesse does DuraMAT data hub have an api?

[adriesse]

@adriesse does DuraMAT data hub have an api?

Yes, but I'm not familiar with it.

https://docs.ckan.org/en/2.8/api/

The data sets we put on Duramat are also listed here:

https://pvpmc.sandia.gov/datasets/

Although I see the spectral response set is missing...

[echedey-ls]

Thanks everyone for all the input.

For the dataset, which is ~328 kB, there are several modules with each of their cells QE characterized ([ 60. 76. 384.] unique number of cells QE per module). And just one SR per module.

I think it makes sense to just add the SR mean of each technology (mono, poly, hit). In addition, SR <> QE conversion functions (nice point @markcampanelli).

Maybe a way of retrieving QEs would be to use this mean SRs and a conversion func since QE data is not complete for all wavelengths. The following graphs plot each of the SRs of each module by cell tech, in order to determine whether it makes sense to use the mean. Code down below if you would like to analyse the data locally.

MonoSi

PolySi

Hit

Code to read dataset & replicate graphs

Remember to change the path (line 23) if running locally.

import pvlib
import h5py
import pandas as pd
import matplotlib.pyplot as plt

from pathlib import Path

keys = [
    "Canadian_270_poly",
    "Canadian_275_mono",
    "Itek_360_mono",
    "Jinko_260_poly",
    "LG_320_mono",
    "LG_400_mono",
    "Mission_300_mono",
    "Panasonic_325_hit",
    "Qcells_280_poly",
    "Qcells_300_mono",
    "Solaria_400_mono",
    "Trina_260_poly",
]

sr_dataset_path = Path(pvlib.__path__[0]).joinpath(
    "data", "Duramat_spectral_responses.nc"
)
datafile = h5py.File(sr_dataset_path, "r")

keys_grouped = {
    "poly": [
        "Canadian_270_poly",
        "Jinko_260_poly",
        "Qcells_280_poly",
        "Trina_260_poly",
    ],
    "mono": [
        "Canadian_275_mono",
        "Itek_360_mono",
        "LG_320_mono",
        "LG_400_mono",
        "Mission_300_mono",
        "Qcells_300_mono",
        "Solaria_400_mono",
    ],
    "hit": ["Panasonic_325_hit"],
}

# ncells = pd.Series(index=datafile.keys())

for cell_type, module_list in keys_grouped.items():
    wavelength = pd.DataFrame()
    sr = pd.DataFrame()
    print(cell_type)
    for module_name in module_list:
        mod = datafile[module_name]
        wavelength[module_name] = mod["wavelength"]
        sr[module_name] = mod["sr"]
        plt.plot(wavelength[module_name], sr[module_name], label=module_name)
        # ncells[module_list] = mod["qe"].shape[0]

    # print("Std Wv")
    # print(wavelength.std())
    # print("Mean SR")
    print("Stdev SR")
    sr_std = sr.std(axis=1)
    plt.plot(
        wavelength[module_name],
        sr.mean(axis=1),
        label="Mean",
        c="k",
        linestyle="--",
    )
    print(sr_std)
    print("Max std:", sr_std.max())
    print("Max percent std:", (sr_std / sr.mean(axis=1)).max())
    plt.grid()
    plt.legend()
    plt.show()
    plt.savefig(f"duramat_sr_agg_{cell_type}.png")
    plt.cla()
    plt.clf()

# print("ncells")
# print(ncells.unique())

[adriesse]

It's unfortunate that you didn't find my programming example, but this is somewhat understandable since it is bundled with the spectral data here:

https://datahub.duramat.org/en/dataset/ghi-spectra-abq

I recommend using xarray to read and explore the netcdf files. You'll find lots of metadata embedded in the files that way also.

[echedey-ls]

Following my GSoC mailing list thread, I've opened two issues (#2037 & #2040) to break down the topics that have arisen here.

I've also opened three PRs to propose additions for each one of them. Feel free to check them out and point any suggestions :)

[kandersolar]

I do not know the details, but I hear that the AM 1.5 reference spectra are currently under revision, or perhaps have already been revised. The NREL NSRDB team had a poster this week at the PVPMC workshop about it. Unfortunately I did not get a chance to discuss it with them in person. Anyway, I bring it up as motivation for considering naming and designing these functions in such a way that accommodates multiple editions of the reference spectra.

[adriesse]

accommodates multiple editions of the reference spectra.

You could include some historical ones too.

[echedey-ls]

I see there are many standards out there, the most recent one I believe is ASTM-G173-23.
I don't have access to any of them except for the freely available ones in the NREL page, which are two AM0 apart from ASTM-G173-03. Historical standards are behind a paywall too, at least everywhere I checked. Maybe you can point me to some open resource?

In any case, I will change the implementation to be easy-to-update in the future without breaking changes.

[kandersolar]

Yes, I believe ASTM G173-23 is the update to which I am referring. I will ask if NREL plans to make the revised spectra public.

[kandersolar]

I heard back that NREL intends to update their webpage with the latest tables, but it may not happen for a while. I think we should proceed with the older tables for now and plan to include the newer tables whenever they become publicly available.

[echedey-ls]

Nice. Thanks for reporting back. Tomorrow I'll come up with an implementation to allow selecting a standard by name.

[adriesse]

Just in case it is of interest, there are some additional spectra defined in ISO-9060 for pyranometer classification available here:

https://standards.iso.org/iso/9060/ed-2/en/

[echedey-ls]

Thanks for the reference @adriesse

I don't quite understand the zip, but I believe they only provide the error of different instruments. I'm not sure if we want to include them here.

I've proposed an extendable function at #2039. Feel free to check it out and drop your suggestions.

[adriesse]

Thanks for the reference @adriesse

I don't quite understand the zip, but I believe they only provide the error of different instruments. I'm not sure if we want to include them here.

I've proposed an extendable function at #2039. Feel free to check it out and drop your suggestions.

There are definitely spectra in there. It might be worth a look how you might include them, even if in the end you don't.

[echedey-ls]

Sorry @adriesse , you are completely right. I thought usual spectrum had values x10 times bigger. Here is a plot with the files:

Script

from pvlib.spectrum import get_am15g
import pandas as pd
from pathlib import Path
import matplotlib.pyplot as plt

PATH_TO_DATA_FOLDER = Path(".")

fig, ax = plt.subplots()
fig.suptitle("Spectra files ISO9060")

files = PATH_TO_DATA_FOLDER.rglob("ext_smarts295.*.txt")
for file in files:
    name = file.name.split(".")[1]
    data = pd.read_csv(
        file, skiprows=1, sep=" ", names=["wv", "extra", "GHI", "beam", "diffuse"], header=0
    )
    # header:
    # Wvlgth Extraterrestrial_spectrm Global_horizn_irradiance Beam_normal_+circumsolar Difuse_horiz-circumsolar
    ax.plot(data["wv"], data["beam"] + data["diffuse"], label=name)

am15g = get_am15g()
ax.plot(am15g.index, am15g, label="pvlib AM1.5G")
ax.legend()
plt.show()

If you consider any of these spectra of interest, please let me know.

[adriesse]

If you consider any of these spectra of interest, please let me know.

I'm neutral. Just wanted you to be aware of them. If anyone want these, please add a thumbs-up...

[AdamRJensen]

I'm in favor of keeping this PR rather simple and only adding one or two of the standard spectrum for now and nailing the function structure. Then I will be easy to add other spectrums in a follow up PR.