This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
Bad gateway error when trying to access TROPOMI files #594
Comments
|
@zfasnacht, have you accepted the EULA? If not, that might be the problem. To accept the EULA, open the URL mentioned in the error message in a browser, which should redirect you to login to Earthdata Login, and then to an EULA (End User License Agreement) page, where you can check to box at the bottom of the page and click the Agree button. Once you do that, you should get past this problem, assuming you haven't already accepted the EULA, and assuming you're using the same credentials via earthaccess as you do when you accept the EULA. |
|
Well it's not happening for the first file, so I'm not sure that would be the case. It reads a few files, then randomly that error occurs. Might read 2 files ok, might read 7, seems to be random. |
|
I did go to that link, logged in, and it downloaded fine which I think suggests I've already accepted the EULA. |
|
Can you share your code? Just enough to show how you're using earthaccess. |
|
Of course, thanks for the help! import earthaccess
import h5py
start_date = '2024-05-22 00:00:00'
end_date = '2024-05-22 23:59:59'
def grab_pace_data(start_date,end_date):
earthaccess.login(persist=True)
results = earthaccess.search_data(short_name = 'S5P_L2__NO2____HiR',cloud_hosted=True,temporal=(start_date,end_date),count=20,bounding_box=(-180,-90,180,90))
trop_no2_files = earthaccess.open(results)
for filename in trop_no2_files:
print(filename.full_name)
f = h5py.File(filename,'r')
data_group = '/PRODUCT/SUPPORT_DATA/DETAILED_RESULTS/'
product_group = '/PRODUCT/'
no2_scd = f[data_group+'nitrogendioxide_slant_column_density'][0]
no2_strat = f[data_group+'nitrogendioxide_stratospheric_column'][0] |
|
This might have to do with the underlying async and multi-threading happening under the covers with the To see if my hunch is correct, try doing the following instead, and let me know if this avoids the issue. import earthaccess
import h5py
start_date = '2024-05-22 00:00:00'
end_date = '2024-05-22 23:59:59'
def grab_pace_data(start_date,end_date):
data_group = '/PRODUCT/SUPPORT_DATA/DETAILED_RESULTS/'
product_group = '/PRODUCT/'
earthaccess.login(persist=True)
results = earthaccess.search_data(short_name = 'S5P_L2__NO2____HiR',cloud_hosted=True,temporal=(start_date,end_date),count=20,bounding_box=(-180,-90,180,90))
for result in results:
with (
earthaccess.open([result])[0] as trop_no2_file,
h5py.File(trop_no2_file) as f
):
print(trop_no2_file.full_name)
f = h5py.File(trop_no2_file,'r')
no2_scd = f[data_group+'nitrogendioxide_slant_column_density'][0]
no2_strat = f[data_group+'nitrogendioxide_stratospheric_column'][0]This will cause each file to be open and closed in sequence. The way most people use Although I wouldn't normally suspect a "Bad Gateway" error to be a result of such potential caching/threading conflicts, I've certainly seen misleading error messages before. Alternatively, it might literally be a flaky server causing intermittent "Bad Gateway" errors. Regardless, I still recommend the "safer" file handling approach I gave above. If it doesn't fix this specific problem, it should at least avoid other potentially gnarly behavior. |
|
Oh geez, that's a great point. I actually am normally careful about closing files but it looks like I did miss that so you might be very right. I'll give that a try. Thanks for the help! |
|
So I'm making sure I close the file now but it still seems like after I read 1-2 files I get the Bad Gateway error Any other possible suggestions to improve this? |
|
It's a different file every time, right? You mentioned that this is random. Perhaps a retry mechanism which "backs off" a little bit by waiting an increasing number of seconds (to a limit) with each retry would help work around this. It's possible this explanation from @chuckwondo is the issue:
GES DISC may appreciate a heads up about this or be better able to help troubleshoot. |
|
I'll give the retries a test. Problem is that it's happening so frequently that I'm not sure how much that will help. It seemed today like I actually went for 30mins to an hour without being able to access a single file. I sent a message to the contact info for earthdata but as you suggest I'll also reach out to the GES DISC folks. Thanks again for all the help! |
|
We're happy to help any time! I'm going to close this issue since we have a new issue to track the need for us to implement retries internally, but if you feel there's more to talk about or that the issue should be re-opened, please feel free to continue to post here. |
|
@zfasnacht I was also having this issue downloading large amounts of TEMPO data (though this data is probably held on different servers than TROPOMI data), and I came across this issue. The 502 Bad Gateway error would occur randomly with or without using the earthaccess API (e.g. with curl also), so it seems to be a server issue. I reduced the number of threads in the earthaccess.download() function to potentially help any overloading. The 502 Bad Gateway errors still persisted but were less frequent. |
|
It's clear there's more to discuss here! I'm going to re-open and convert this to a discussion. |
This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
I'm trying to use the earthaccess tool to read TROPOMI files that are in the GES DISC cloud but I'm getting the following error quite frequently
Any idea how to avoid this from happening other then a simple try/except?
The text was updated successfully, but these errors were encountered: