S5P-PAL Product Search

Interactive download

S5P-PAL product files can be selected and downloaded using the SpatioTemporal Asset Catalog (STAC) browser interface built on top of our implementation of the STAC Collection Specification.

Because STAC is an open standard, you are not restricted to using this S5P-PAL-hosted GUI. If you have access to another STAC browser or viewer, you can point it directly towards the S5P-PAL 'sentinel-5p' catalogue at:

https://data-portal.s5p-pal.com/cat/sentinel-5p/catalog.json

(or any of the sub-level collection catalogues you will see displayed on the screen when browsing) and it should work out of the box.

Programmatic access

Similarly, you can use a STAC library, such as the PySTAC Client for Python, to programmatically access the S5P-PAL product catalogue and, for example, obtain download links for product files that way. (There are similar packages and tools for other languages.)

S5P-PAL implements the STAC Item Search API, exposing a search endpoint for querying items in the catalogue based on geospatial or temporal criteria. This search endpoint is not available through the current version of the browser interface, so in order to make use of it doing so programmatically is, for now, the only way.

Item Search Examples

In this section we give some basic examples to help you on your way, using the aforementioned PySTAC Client library for Python. PySTAC Client requires that you also have the lower-level PySTAC library installed.

The following is a simple script that will print some information about the most recently generated product in the catalogue, and then download it. (This script uses the Python Requests library to handle the actual download.)

from pystac import Collection
from pystac_client import ItemSearch
import requests
import hashlib


def get_most_recent_product():
    s5pcoll = Collection.from_file(
        "https://data-portal.s5p-pal.com/cat/sentinel-5p/catalog.json"
    )
    endpoint = s5pcoll.get_single_link("search").target

    items = ItemSearch(
        endpoint,
        sortby="-properties.archive_date",
        max_items=1,
    ).get_items()
    item = list(items)[0]

    download_url = item.assets["download"].href
    product_filename = item.properties["physical_name"]
    product_hash = item.properties["hash"]

    print(f"Downloading {product_filename}...")
    r = requests.get(download_url)
    with open(f"./{product_filename}", "wb") as product_file:
        product_file.write(r.content)
    file_hash = "md5:" + hashlib.md5(open(product_filename, "rb").read()).hexdigest()
    print("Checking hash...")
    assert file_hash == product_hash
    print("Product was downloaded correctly")


if __name__ == "__main__":
    get_most_recent_product()
Downloading S5P_PAL__L2__NO2____20211114T073346_20211114T091516_21181_02_020301_20211215T133946.nc...
Checking hash...
Product was downloaded correctly

As you can see in the code, we configure an instance of the ItemSearch class to query the endpoint. The various filtering parameters you can supply are listed in the PySTAC API reference, but of particular interest are datetime and intersects. The former allows searching for product within a time interval, the second allows searching for products that intersect with a polygon. For example:

timefilter = "2018-05-01"
items = ItemSearch(endpoint, datetime=timefilter).get_items()
print(f"found {len(list(items))} products for time period {timefilter}")

geofilter = {
    'type': 'Polygon',
    'coordinates': [[[6.42425537109375, 53.174765470134616], [7.344360351562499, 53.174765470134616], [7.344360351562499, 53.67393435835391], [6.42425537109375, 53.67393435835391], [6.42425537109375, 53.174765470134616]]]
}
items = ItemSearch(endpoint, datetime=timefilter, intersects=geofilter).get_items()
print(f"found {len(list(items))} products for time period {timefilter} when geofiltered")
found 15 products for time period 2018-05-01
found 2 products for time period 2018-05-01 when geofiltered

The first ItemSearch gives us the 15 NO2 products generated for May 01, 2018. The second ItemSearch narrows the search down to those products whose footprint intersects with the specified region.

The call to get_items() returns objects of type pystac.item.Item objects, which can then be further examined to retrieve, for example, the product download links:

print("Download links:")
for item in list(items):
    url = item.assets['download'].href
    print(f"  {url}")
Download links:
  https://data-portal.s5p-pal.com/cat/sentinel-5p/download/06d39838-5903-4d4c-a39e-91506d7f3011
  https://data-portal.s5p-pal.com/cat/sentinel-5p/download/b57cfe5b-d6c6-4200-9f55-ee89658de8d2

Item Search extensions

The S5P-PAL STAC search implementation also supports the STAC Sort and the STAC Filter fragments. These are OpenAPI extensions to the basic search query parameters accepted by the Search API.

The Sort fragment introduces the sortby parameter we already used in the 'get_most_recent_product' example above, allowing you to define fields by which to sort the query results (in that case: descending by archive_date).

The Filter fragment provides a mechanism for searching based on item attributes. For example:

items = list(ItemSearch(
    endpoint,
    filter="s5p:file_type='L2__NO2___' and (s5p:orbit>21176 or s5p:orbit=2841)",
).get_items())

will select only those products matching that particular filter expression.

Finally, it it also possible for the filter to be specified as a JSON dictionary (as detailed in the Filter Fragment documentation) rather than as a text string. Our previous example would then become:

items = list(ItemSearch(
    endpoint,
    filter=
    {
        "op": "and", "args": [
            { "op": "=", "args": [ {"property": "s5p:file_type"}, "L2__NO2___" ] },
            {
                "op": "or", "args": [
                    { "op": ">" , "args": [ {"property": "s5p:orbit"}, 21176 ] },
                    { "op": "=" , "args": [ {"property": "s5p:orbit"}, 2841 ] }
                ]
            }
        ]
    }
).get_items())

Note that in the text example the single quotes around string values such as the file type specification L2__NO2___ are mandatory, where in the JSON version they should not be used. Similarly, in the text version, operators such as and and or are case-insensitive, whereas in the JSON version they must be lowercase.

Browsing examples

It is also possible to use just the PySTAC library to programmatically query the Catalog endpoint. Typically this is a less interesting use case, because traversing the catalogue tree is easier done by using the interactive browser, but it is certainly possible.

For example, the NO2 Catalog can also be retrieved via the root catalog, and subsequently drilled further down into to obtain its Items:

from pystac import Collection


def browse_to_no2_items():

    coll = Collection.from_file("https://data-portal.s5p-pal.com/cat/catalog.json")
    no2coll = coll.get_child(id="sentinel-5p").get_child(id="S5P_L2__NO2___")
    daycoll = no2coll.get_child(id="2020").get_child(id="09").get_child(id="27")
    print("Collection ID:", daycoll.id)
    print("Collection description:", daycoll.description)
    print("Spatial extent:", daycoll.extent.spatial.bboxes)
    print("Temporal extent:", daycoll.extent.temporal.intervals)

    items = daycoll.get_items()
    print("Items in collection:")
    for index, item in enumerate(items, 1):
        product_filename = item.properties["physical_name"]
        download_url = item.assets["download"].href
        print(f"  product {index:2}: {product_filename}")
        print(f"         url: {download_url}")


if __name__ == "__main__":
    browse_to_no2_items()

This will output all the products for Sep 27, 2020:

Collection ID: 27
Collection description: Collection for product type S5P_L2__NO2___ (2020-09-27)
Spatial extent: [[-180.0, -90.0, 180.0, 90.0]]
Temporal extent: [[datetime.datetime(2020, 9, 27, 0, 0, tzinfo=tzutc()), datetime.datetime(2020, 9, 28, 0, 0, tzinfo=tzutc())]]
Items in collection:
  product  1: S5P_PAL__L2__NO2____20200927T013649_20200927T031818_15318_01_020301_20211111T173014.nc
         url: https://data-portal.s5p-pal.com/cat/sentinel-5p/download/7b365bfe-091a-4fe4-9586-ac126ceee412
  product  2: S5P_PAL__L2__NO2____20200927T031818_20200927T045948_15319_01_020301_20211111T173050.nc
         url: https://data-portal.s5p-pal.com/cat/sentinel-5p/download/43833a93-cbe3-4da9-a1b1-319546dbac6f
  product  3: S5P_PAL__L2__NO2____20200927T045948_20200927T064117_15320_01_020301_20211111T173122.nc
         url: https://data-portal.s5p-pal.com/cat/sentinel-5p/download/5198f9c5-e469-45c8-98ae-3d1b1c213f76
  product  4: S5P_PAL__L2__NO2____20200927T064117_20200927T082247_15321_01_020301_20211108T042747.nc
         url: https://data-portal.s5p-pal.com/cat/sentinel-5p/download/21ca8ec1-99c6-45d3-9815-0b81cf6d2421
  product  5: S5P_PAL__L2__NO2____20200927T082247_20200927T100416_15322_01_020301_20211111T173232.nc
         url: https://data-portal.s5p-pal.com/cat/sentinel-5p/download/2d1ea8d0-c93e-43c6-b235-4a0c8bbbe29f
  product  6: S5P_PAL__L2__NO2____20200927T100416_20200927T114546_15323_01_020301_20211111T173238.nc
         url: https://data-portal.s5p-pal.com/cat/sentinel-5p/download/5f809755-7582-48c7-a741-d949b9fbe78b
  product  7: S5P_PAL__L2__NO2____20200927T114546_20200927T132715_15324_01_020301_20211111T173240.nc
         url: https://data-portal.s5p-pal.com/cat/sentinel-5p/download/65b41cc8-c83d-472c-b33d-9a1b7ca6f048
  product  8: S5P_PAL__L2__NO2____20200927T132715_20200927T150844_15325_01_020301_20211111T173244.nc
         url: https://data-portal.s5p-pal.com/cat/sentinel-5p/download/25a69539-4e9b-49bd-8e76-baa0b2eb4f2f
  product  9: S5P_PAL__L2__NO2____20200927T150844_20200927T165014_15326_01_020301_20211111T173333.nc
         url: https://data-portal.s5p-pal.com/cat/sentinel-5p/download/de05f869-a27d-4101-b4f7-0ec588e88cd6
  product 10: S5P_PAL__L2__NO2____20200927T165014_20200927T183143_15327_01_020301_20211111T173411.nc
         url: https://data-portal.s5p-pal.com/cat/sentinel-5p/download/94589a9e-368b-4cee-ba07-a8ea32dcdaf3
  product 11: S5P_PAL__L2__NO2____20200927T183143_20200927T201312_15328_01_020301_20211111T173412.nc
         url: https://data-portal.s5p-pal.com/cat/sentinel-5p/download/6541ab60-e3eb-4130-9d96-e4bee891bcdb
  product 12: S5P_PAL__L2__NO2____20200927T201312_20200927T215442_15329_01_020301_20211111T173416.nc
         url: https://data-portal.s5p-pal.com/cat/sentinel-5p/download/0a4f0054-863c-4a54-b95e-61a650675921
  product 13: S5P_PAL__L2__NO2____20200927T215442_20200927T233611_15330_01_020301_20211111T173459.nc
         url: https://data-portal.s5p-pal.com/cat/sentinel-5p/download/fc0e8e49-4225-4ceb-ab8e-409cf13b5334
  product 14: S5P_PAL__L2__NO2____20200927T233611_20200928T011741_15331_01_020301_20211111T173509.nc
         url: https://data-portal.s5p-pal.com/cat/sentinel-5p/download/a999b912-678f-42fc-bb1a-d9cbf9f83843

Limitations

The PySTAC Client package also contains a command-line utility (as well as some other programmatic classes and functions) that are not currently compatible with S5P-PAL's catalogues. This may change in the future, but for now the method described above, using ItemSearch and an explicitly created endpoint are the way to approach searching. This caveat is of course specific to PySTAC and may not apply to other libraries and tools.

Support

This service is provided as part of the Sentinel-5P Product Algorithm Laboratory (S5P-PAL) and contains modified Copernicus Sentinel data processed by S[&]T.

Questions regarding this service can be send to the ESA EO Support Helpdesk.