Test pyroSAR tool

[echeipesh]

We'd like to consume Sentinel-1 GRD from AWS public dataset: https://registry.opendata.aws/sentinel-1/
However, the scenes stored there are geo-located but not rectification.

It's been suggested that pyrosar tool may be useful to finish the processing stages: https://pyrosar.readthedocs.io/en/latest/general/processing.html
This issue is to explore and test that tool. We are still pretty new to SAR processing so I'm not totally clear on what steps are required. Since GEE was used for this processing by Sen1Floods11 paper it makes sense to compare the capability of this tool in regards to SAR processing on GEE and try to line up relevant functionality from the paper.

• Document what what processing steps
• Install and run the pyrosar on some actual sentinel-1 scenes
• Compare SAR processing functionality to provided that used by GEE.
• Compare output with GEE output.

[CloudNiner]

Releasing assignment because I'm unblocked on #38 and have some outstanding tasks to finish for it.

[CloudNiner]

Below is a rough draft of my notes for what I've discovered so far. Key takeaways from the notes:

• PyroSAR is likely best installed in a Docker container via Conda. Woo. Manual installation instructions were fraught with issues
• Cannot use PyroSAR to directly read downloads of the S1 data in the AWS bucket. PyroSAR works with SNAP files, which AWS are some "cloud optimized" transformation of
  • I don't know yet if ESA SNAP toolbox can directly read the AWS bucket
  • There are two utilities that could potentially solve this problem. One only works with S2, and the other requires a copernicus APIHub login and is suuuuper rough around the edges
• The notes include a bunch of links to references that describe more about the S1 modes, product types, levels, etc

PyroSAR R/D

Installation

Attempted manual installation in Docker. PyroSAR doc instructions are incomplete and I still had issues with incompatible numpy versions after installation. Old numpy version 1.13 was installed

Attempted conda installation in Docker. This worked — image is HUGE though, 3.3GB

FROM continuumio/miniconda3

RUN conda create -n env python=3.7
RUN conda install --yes --channel conda-forge pyroSAR ipython
RUN echo "source activate env" > ~/.bashrc
ENV PATH /opt/conda/envs/env/bin:$PATH

WORKDIR /app

ENTRYPOINT [ "ipython" ]

version: '3'
services:
  pyrosar:
    build: .
    image: pyrosar
    volumes:
      - .:/app

Sentinel 1 Schema

Mission Identifier: S1A or S1B, the instrument that captured the scene

TONS of general info about Synthetic Aperture Radar (SAR) and how it works in this NASA PDF

Mode:
- S1 - S6: SM (Strip Map) - Only used for small islands and on request for extraordinary events such as emergency management
- User Guides - Sentinel-1 SAR - Stripmap - Sentinel Online
- IW (interferometric Wide): Primary mode over land, VV+VH dual polarization
- User Guides - Sentinel-1 SAR - Interferometric Wide Swath - Sentinel Online
- EW (Extra Wide): Primarily used for wide area coastal monitoring including ship traffic, oil spill and sea-ice monitoring
- User Guides - Sentinel-1 SAR - Extra Wide Swath - Sentinel Online
- WV (Wave): Primary mode over open ocean, VV single polarization
- User Guides - Sentinel-1 SAR - Wave - Sentinel Online

Processing Level:
- Level 0: Consists of compressed and unfocused SAR raw data.
- User Guides - Sentinel-1 SAR - Level-0 - Sentinel Online
- Level-0 products are the basis from which all other high level products are produced.
- Level 1: The products intended for most data users
- User Guides - Sentinel-1 SAR - Level-1 - Sentinel Online
- Level 2: Geolocated geophysical products derived from Level-1
- User Guides - Sentinel-1 SAR - Level-2 - Sentinel Online
For each mode it is possible to produce products at SAR Level-0, Level-1 SLC, Level-1 GRD and Level-2 OCN

Product Type:
- RAW: level 0 product
- SLC: Single Look Complex - this is a level 1 product
- “focused SAR data, geo-referenced using orbit and attitude data from the satellite, and provided in slant-range geometry. Slant range is the natural radar range observation coordinate, defined as the line-of-sight from the radar to each reflecting object.”
- User Guides - Sentinel-1 SAR - Level-1 - Sentinel Online
- GRD: Ground Range Detected - this is a level 1 product
- “focused SAR data that has been detected, multi-looked and projected to ground range using the Earth ellipsoid model WGS84”
- User Guides - Sentinel-1 SAR - Level-1 - Sentinel Online
- OCN: Ocean (OCN) products for wind, wave and currents applications may contain the following geophysical components derived from the SAR data
- User Guides - Sentinel-1 SAR - Level-2 - Sentinel Online

Resolution Class, used by GRD product type only:
- F: full resolution
- H: high resolution
- M: medium resolution
- _: Not applicable to product type
Processing Level: 0, 1 or 2
Product Class: Standard (S) or Annotation (A)
Polarization:
- SH: Single HH polarization
- SV: Single VV polarization
- DH: Dual HH + HV polarization
- DV: Dual VV + VH polarization
Start Date
End Date
Absolute Orbit Number
Mission Data Take ID
Product Unique ID: hexadecimal string generated by computing CRC-16 on the manifest file using CRC-CCITT

On SentinelHub, scenes are available as download in SAFE format: User Guides - Sentinel-1 SAR - SAFE Specification - Sentinel Online
AWS writes a different “cloud-optimized” format, but the AWS README for the data doesn’t specify how its different: Sentinel-1 GRD dataset

Getting Data

AWS

Attempted download to local dir and read with PyroSAR:

aws s3 cp \
  s3://sentinel-s1-l1c/GRD/2018/2/15/IW/DV/S1A_IW_GRDH_1SDV_20180215T000433_20180215T000502_020610_02348A_6782/ \
  ./S1/S1A_IW_GRDH_1SDV_20180215T000433_20180215T000502_020610_02348A_6782/ \
  --request-payer --recursive

import os
from pyroSAR import identify
name = 'S1A_IW_GRDH_1SDV_20180215T000433_20180215T000502_020610_02348A_6782'
scene = identify(os.path.join('S1', name))
# FAILS: RuntimeError: data format not supported

# Change name dir to have a `.SAFE` suffix
from pyroSAR.drivers import SAFE
name = 'S1A_IW_GRDH_1SDV_20180215T000433_20180215T000502_020610_02348A_6782.SAFE'
scene = SAFE(os.path.join('S1', name))
""" FAILS:
/opt/conda/lib/python3.7/site-packages/pyroSAR/drivers.py in scanMetadata(self)
   1403 
   1404         annotations = self.findfiles(self.pattern_ds)
-> 1405         with self.getFileObj(annotations[0]) as ann_xml:
   1406             ann_tree = ET.fromstring(ann_xml.read())
   1407 

IndexError: list index out of range
"""

Need this to convert data in AWS bucket to pyrosar compatible format? YES

Python sentinelhub library provides this for sentinel 2 data: Downloading satellite data from AWS with command line — Sentinel Hub 3.0.5 documentation
But not for sentinel 1 data: Support for Sentinel-1 Download · Issue #89 · sentinel-hub/sentinelhub-py · GitHub
- They’re taking PRs though, so we could implement it if we had the time

• Using AWS S1 GRD data for SAR processing - Sentinel@AWS - Forum@Sentinel Hub
• Processing on AWS · Issue #107 · johntruckenbrodt/pyroSAR · GitHub
• GitHub - rodolfolotte/aws_imagery_pack: Python library for searching and downloading orbital images using AWS API

Processing

GEE Tutorial: Sentinel-1 Algorithms  |  Google Earth Engine  |  Google Developers

PyroSAR

PyroSAR is a few things, but the interesting methods are a wrapper around the ESA Sentinel Application Platform (SNAP) toolbox: SNAP | STEP

I think this is the method we’d want to use to perform sentinel 1 preprocessing if we were using PyroSAR: pyroSAR.snap.util.geocode

[CloudNiner]

Here's the notes from my brief email conversation with Ganesh.

TL;DR:

• He doesn't know of any converters that would speed AWS public sentinel1 ingestion into pyroSAR
• If what we need is preprocessed sigma0 sentinel1 scenes, we should look at cost/benefit to just paying to access them from sentinelhub
• SNAP processing would require relatively large machines, its RAM hungry (16+GB)
• They pull sentinel GRD scenes from ASF Vertex instead of Copernicus or sentinelhub

We use PyroSAR to process data in the native format. We don't use Sentinelhub at this time in our processing pipeline. The GRD granules are directly downloaded from the ASF Vertex site (similar to Copernicus hub): https://asf.alaska.edu/api/. We query for the available GRD products on the ASF vertex site and use PyroSAR (+ SNAP through its snappy interface in the backend) to generate the Sigma0 products. Just a note on SNAP: in our experience, we would need a decent amount of RAM (16GB or more is optimal) to generate the Sigma0 products. The Java code in SNAP is not very optimum implementation. There is a provision to use GAMMA instead of SNAP but GAMMA needs a license. Given the number of specific sites we need to monitor in a continuous time-fashion for our project, we elected to use the freely available data and our own processing pipeline to keep the net costs reasonable for the customer. We mostly use Sentinel-1. We use Sentinel-2 occassionally for visual inspection. For most of our sites, Sentinel-2 is not useful due to persistent cloud cover, etc. We, unfortunately, don't have any format converters implemented on our end.

If you prefer to not deal with the processing steps to generate Sigma0 products, I believe Sentinelhub already has them available through their AWS bucket. I am aware of the following python lib from Sentinelhub that lets you access the products. https://sentinelhub-py.readthedocs.io/en/latest/. There is a Sentinelhub fee that gives you the keys to download the pre-generated products like Sigma0, Gamma0, etc. This can save you a lot of time in creating your own Sigma0 products etc. By default, Sentinelhub will offer gamma0 products, but for flood detection, you are probably better off using Sigma0 data. There is a small setting change that needs to be made in your Sentinelhub account to display/download Sigma0 products. I can send some screenshots if you choose to use Sentinelhub in your work.

[CloudNiner]

@jcahail @echeipesh In light of the above, I’m going to call the pyroSAR r/d task done in the absence of specific threads to pull on. A decision on whether to just pay sentinel hub extra for sentinel 1 GRD sigma0 or try to preprocess it ourselves is outside scope of my knowledge but I want to flag it here. Whoever tackles that preprocessing task should be aware of it as an option to investigate based on time & effort concerns.