Is anyone else experiencing 'USGS not currently responding to requests'?

When catching the error it is giving me a 403 Forbidden.

My user name and password are working in Earth Explorer web interface when I try download the exact file using the link LANDSAT-Download generates.

I have logged out of Earth Explorer before using this script as well.

Good evening Olivier,

Thank you very much for sharing this script! I'm a first-time user, and I've been running into an HTTP Error 503. At the moment, I'm just testing things out by using your README example, with no proxy and my own usgs.txt file:

download_landsat_scene.py -o scene -b LC8 -d 20130127 -s 043034 -u usgs.txt --output /home/miriam/Documents/test/

The full error is:

None None Traceback (most recent call last): File "download_landsat_scene.py", line 594, in main() File "download_landsat_scene.py", line 412, in main connect_earthexplorer_no_proxy(usgs) File "download_landsat_scene.py", line 79, in connect_earthexplorer_no_proxy f = urllib2.urlopen(request) File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen return _opener.open(url, data, timeout) File "/usr/lib/python2.7/urllib2.py", line 410, in open response = meth(req, response) File "/usr/lib/python2.7/urllib2.py", line 523, in http_response 'http', request, response, code, msg, hdrs) File "/usr/lib/python2.7/urllib2.py", line 442, in error result = self._call_chain(*args) File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain result = func(*args) File "/usr/lib/python2.7/urllib2.py", line 629, in http_error_302 return self.parent.open(new, timeout=req.timeout) File "/usr/lib/python2.7/urllib2.py", line 410, in open response = meth(req, response) File "/usr/lib/python2.7/urllib2.py", line 523, in http_response 'http', request, response, code, msg, hdrs) File "/usr/lib/python2.7/urllib2.py", line 448, in error return self._call_chain(*args) File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain result = func(*args) File "/usr/lib/python2.7/urllib2.py", line 531, in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) urllib2.HTTPError: HTTP Error 503: Service Unavailable

Generally, I think a 503 error would signal some issue with the webpage itself, but I can sign in just fine using a browser, and a quick check with curl (curl -I "https://ers.cr.usgs.gov/login") says "HTTP/1.1 200 OK."

Do you know what the source of this issue may be? Many thanks, in advance. I really appreciate your help.

First pull request!

For some reason there was an extra space in some of the code so I removed those in the area I was working in. I'm not sure if this works for proxy but it worked fine for no_proxy.

Brad-Batess-MacBook-Pro:Landsat_Data bradbates$ python download_landsat_scene.py -o scene -b LC8 -d 20150526 -f 20150601 -s 029033 -u usgs.txt --output /Users/bradbates/Desktop/Landsat_Data/Landsat_8 None None Establishing connection to Earthexplorer... Traceback (most recent call last): File "download_landsat_scene.py", line 564, in main() File "download_landsat_scene.py", line 384, in main connect_earthexplorer_no_proxy(usgs)
File "download_landsat_scene.py", line 58, in connect_earthexplorer_no_proxy f = opener.open("https://earthexplorer.usgs.gov/login/", params) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 410, in open response = meth(req, response) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 523, in http_response 'http', request, response, code, msg, hdrs) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 448, in error return self._call_chain(_args) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 382, in _call_chain result = func(_args) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 531, in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) urllib2.HTTPError: HTTP Error 404: Not Found

I added the following options:

• z - to unzip downloaded tgz files. Uses the tartool (https://tartool.codeplex.com/)
• b - which lets you select the product you're looking for (LT5, LE7 or LC8)
• c - which lets you define a cloud cover limit for the downloaded images. If an image is found with clouds below the limit the script stops looking for more images within the time interval. If not, it doesn't keep the image and tries to find another.

Landsat 5 and Landsat 7 download wasn't working (at least for me). I fixed that.

The connection to earthexplorer was established within the while loop, so it was repeating it for each iteration. I moved that block to before the while loop, making the process much faster.

something changed in the USGS interface, not allowing the proxying "urllib2.URLError: <urlopen error Tunnel connection failed: 403 Forbidden>" Any idea on why this occurs?

Hi there,

I am getting an error and don't know the actual issue? The query I run is as follows: "E:\xx_2_Python_Automation_Scripts\TEMP\Automated_Download_LANDSAT\LANDSAT-Download-master>download_landsat_scene.py -o scene -b LC8 -d 20161001 -s 170073 -u E:\xx_2_Python_Automation_Scripts\TEMP\Automated_Download_LANDSAT\LANDSAT-Download-master\usgs.txt --output E:\xx_2_Python_Automation_Scripts\TEMP\Automated_Download_LANDSAT\TEST_OUTPUTS"

And the error that comes up is the following: "Searching for images on (julian date): 2016278... http://earthexplorer.usgs.gov/download/4923/LC81700732016278LGN00/STANDARD/EE error : file is in html and not an expected binary file result saved in ./error_output.html product LC81700732016278LGN00 not found http://earthexplorer.usgs.gov/download/4923/LC81700732016278LGN01/STANDARD/EE http://earthexplorer.usgs.gov/download/4923/LC81700732016278LGN02/STANDARD/EE Searching for images on (julian date): 2016294... http://earthexplorer.usgs.gov/download/4923/LC81700732016294LGN00/STANDARD/EE error : file is in html and not an expected binary file result saved in ./error_output.html product LC81700732016294LGN00 not found http://earthexplorer.usgs.gov/download/4923/LC81700732016294LGN01/STANDARD/EE http://earthexplorer.usgs.gov/download/4923/LC81700732016294LGN02/STANDARD/EE"

Not too sure why it is failing to find the results.

Thanks!

Hi Olivier,

after some testing everything works fine for me on Windows as well as on Linux. There remains just one issue regarding the download speed on Windows. Compared to Linux it is much slower (factor 10 ~200KB/s to 2MB/s), even though the Linux OS is implemented on a Virtual Machine on the same Desktop as Windows(so both are connected to the same Network). I am not a python expert, so do you have any clue what this might cause? Thanks,

Fabian

First of all, congratulations on your script. It will certainly be of much help to me. I was able to download LC8 images, but could get images with dates previous to Landsat 8 being operational. Also, suggestion for future improvement: automatically unpack the tgz files... Maybe I can even try to help you with that if you're interested. Thanks

Hi,

I am trying to run the code to download some landsat scenes but get this errors:

Establishing connection to Earthexplorer with proxy...
Traceback (most recent call last):
  File "download_landsat_scene.py", line 592, in <module>
    main()
  File "download_landsat_scene.py", line 410, in main
    connect_earthexplorer_proxy(proxy,usgs)
  File "download_landsat_scene.py", line 37, in connect_earthexplorer_proxy
    data=urllib2.urlopen("https://earthexplorer.usgs.gov/").read()
  File "/root/anaconda3/envs/py2/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/root/anaconda3/envs/py2/lib/python2.7/urllib2.py", line 429, in open
    response = self._open(req, data)
  File "/root/anaconda3/envs/py2/lib/python2.7/urllib2.py", line 447, in _open
    '_open', req)
  File "/root/anaconda3/envs/py2/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/root/anaconda3/envs/py2/lib/python2.7/urllib2.py", line 1241, in https_open
    context=self._context)
  File "/root/anaconda3/envs/py2/lib/python2.7/urllib2.py", line 1198, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [Errno -2] Name or service not known>

I have hanged from http://ers.cr.usgs.gov/ to http://earthexplorer.usgs.gov/ like you said in the post, but always get the same error. Appreciate your help.

Best regards

Thanks for the code! I am first time user but failed to use the script, here is my script and error: python ./download_landsat_scene.py -o scene -s 100062 -b LC8 --dir 12864 --station 'LGN' -d 20180321 -u usgs.txt

and the error: LGN 12864 Searching for images on (julian date): 2018081... https://earthexplorer.usgs.gov/download/12864/LC81000622018081LGN00/STANDARD/EE URL Error: [Errno 8] _ssl.c:480: EOF occurred in violation of protocol https://earthexplorer.usgs.gov/download/12864/LC81000622018081LGN00/STANDARD/EE

in the output file only one log.txt with [] inside.

I tried one of the links in browser (for example:https://earthexplorer.usgs.gov/download/12864/LC81000622018081LGN00/STANDARD/EE) and it works perfect, and the download started after I input my credential. Not sure what I am doing wrong. Any help is great appreciated!!!

import json from landsatxplore.api import API from landsatxplore.earthexplorer import EarthExplorer

Initialize a new API instance and get an access key

usgs_username = "******" usgs_password = "******"

api = API(usgs_username, usgs_password)

Search for Landsat TM scenes

scenes = api.search( dataset='landsat_ot_c2_l1', latitude=30.9, longitude=75.85, #bbox= (70, 28, 80, 32), #(xmin, ymin, xmax, ymax), start_date='2017-01-01', end_date='2017-04-01', max_cloud_cover=10 )

print(f"{len(scenes)} scenes found.")

Process the result

for scene in scenes: print(scene['acquisition_date'].strftime('%Y-%m-%d')) # Write scene footprints to disk fname = f"{scene['landsat_product_id']}.geojson" with open(fname, "w") as f: json.dump(scene['spatial_coverage'].geo_interface, f) print(f)

api.logout()

output

3 scenes found. 2017-03-19 <_io.TextIOWrapper name='LC08_L1TP_148038_20170319_20200904_02_T1.geojson' mode='w' encoding='cp1252'> 2017-03-19 <_io.TextIOWrapper name='LC08_L1TP_148039_20170319_20200904_02_T1.geojson' mode='w' encoding='cp1252'> 2017-02-15 <_io.TextIOWrapper name='LC08_L1TP_148039_20170215_20200905_02_T1.geojson' mode='w' encoding='cp1252'>

Downloading Scenes

ee = EarthExplorer(usgs_username, usgs_password)

ee.download('LC08_L1TP_148038_20170319_20200904_02_T1', output_dir=r'D:\RF_Paper\JAM\MODIS')

ee.logout()

error

EarthExplorerError Traceback (most recent call last) in 2 ee = EarthExplorer(usgs_username, usgs_password) 3 ----> 4 ee.download('LC08_L1TP_148038_20170319_20200904_02_T1', output_dir=r'D:\RF_Paper\JAM\MODIS') 5 6 ee.logout()

~\Anaconda3\lib\site-packages\landsatxplore\earthexplorer.py in download(self, identifier, output_dir, dataset, timeout, skip) 148 data_product_id=DATA_PRODUCTS[dataset], entity_id=entity_id 149 ) --> 150 filename = self._download(url, output_dir, timeout=timeout, skip=skip) 151 return filename

~\Anaconda3\lib\site-packages\landsatxplore\earthexplorer.py in _download(self, url, output_dir, timeout, chunk_size, skip) 90 error_msg = r.json().get("errorMessage") 91 if error_msg: ---> 92 raise EarthExplorerError(error_msg) 93 download_url = r.json().get("url") 94

EarthExplorerError: Download is not available

Hi,

I used this code without problems earlier this year. Now I keep getting the error "Error : CSRF_Token not found”. If I change line 68:
m = re.search(r'<input .?name="csrf_token".?value="(.?)"', data) to m = re.search(r'<input .?name="csrf".?value="(.?)"', data) I don't get the error. But the download still doesn't work.

Yo yo, I'm trying to webscrape the shitttttt out of Yahoo finance and they blocked me with a 503 error, which I'm given to understand means my IP doesn't work? ANYWAY, this is a thing I want to be able to make into an exe and put online someday and I don't wanna be limited by the silly webscraping rules. Is there an IP randomiser module? Would a VPN module work? Or should I try Tor? Help me out please guys, much appreciated! Or perhaps is there a better way for me to get on demand/live data on stocks etc?

After I run this code in Linux Mint 18.3....

python download_landsat_scene.py -o catalog -b LE7 -d 20170401 -f 20171101 -s 191028 -u usgs.txt --output /home/alen/LANDSAT-Download-master/Landsat_download ... I get this output:

None None Verifying catalog metadata files... Searching for images in catalog... No image was found in the catalog with the given specifications! Exiting...

The same output is after using the manual example: python download_landsat_scene.py -o catalog -b LC8 -d 20130127 -s 198030 -u usgs.txt --output /home/alen/LANDSAT-Download-master/Landsat_download

This script is super cool beacuse it saves time in downloading data and has selective attributes put into consideration. however, I am trying to download Landsat5 data and it is just iterating through the archive with no result. Below is the command

python download_landsat_scene.py -o scene -b LT5 -d 20000101 -f 20001231 -c 50 -s 169060 -u usgs.txt --output ~/Documents/GIS/