Original Data: This example shows a dataset that is relatively clean before any filtering is applied. It seems that this file should have been extremely easy to clean.

Seasat – Technical Challenges – 6. Slope Issues

During decoding and cleaning, it was assumed that the time slope of the files would be roughly guided by the Pulse Repetition Interval (PRI) of the satellite, i.e. a Pulse Repetition Frequency (PRF) of 1647 Hz means that 1,647 lines are being transmitted and received per second. This means that the PRI is 0.00060716 msec. Based upon this, then, each 1,000 lines of Seasat data should be equivalent to .60716 seconds.

Alternately, in milliseconds, the time slope for these files should always be 0.60716. It was discovered that this is not the case with much of the actual data, as shown in the following table and graphs:

Line Time Time Diff Calculated Slope
1 13851543
500 13851790 247 0.4950
10000 13856399 4856 0.4856
15000 13858818 7275 0.4850
20000 13861260 9717 0.4859
30000 13866139 14596 0.4865
35000 13868569 17026 0.4865
40000 13870998 19455 0.4864
45000 13873447 21904 0.4868

Seasat Times: PRF = 1647 Hz, so PRI is 0.0006071645 msec. In MSEC, the time slope should always be 0.6071645. Yet, for this datatake, the time slope is consistently only 0.486!

Original Data

Original Data: This example shows a dataset that is relatively clean before any filtering is applied. It seems that this file should have been extremely easy to clean.

Filtered Data

Filtered Data: After the dataset went through the prep_raw.sh procedure, this was the resulting time plot. It is, quite obviously, very wrong.

Comparison

Comparison of Original with Filtered: Although the times look fine in the first (unfiltered) plot, they are wrong for this satellite based upon the known PRI. The ASF cleaning software tried to fix these wrong time values using a known slope of 0.607. This introduced a discontinuity into the data and resulted in incorrect times.

These results, wherein the time slope of the raw data does not match the known PRI of the satellite, were incredibly perplexing. At first, it was assumed that these data could not be processed reliably and were simply categorized into the large time-slope error and wrong-fix error categories.

Analysis of the time slopes in the original unfiltered data only pointed out how extreme the problem really was. Well over 100 files showed slopes that were either less than 0.606 or more than 0.608, with the lowest in the 0.48 range. The highest reliable estimate showed a slope of well over 0.62.

6.1 Slope Issues Explained

Eventually, through conversation with original Seasat engineers at the Jet Propulsion Lab (JPL), it was discovered that the Seasat metadata field MSEC of Day actually contains not only the time of imaging but also the time to transmit data from the spacecraft to the ground station. This adds a variable time offset to the metadata field. Once this was understood, it was readily obvious that using the known PRF as a guide for filtering was an incorrect solution.

Thus, the entire cleaning process was revisited, with all of the codes allowing more relaxed slope values during linear regression. This worked considerably better than the previous cleaning attempt. However, it did not solve the problems entirely.

6.2 Final Results of Data Cleaning

The final set of cleaned Seasat raw swaths was assembled using three main passes through the archives with different search parameters, along with a few files that were fixed on a case-by-case basis. Basically, the final version of the code was run and the results examined for remaining time gaps. Any files with large or many time gaps were reprocessed using different parameters. In the end, 1,346 swaths were cleaned, 2 by hand, 14 from the first pass, 25 from the second pass, and the remainder in the final cleaning pass. These then are the final cleaned Seasat archives for the initial release of ASF’s Seasat products.

Date Total Datasets Dataset with Time Gaps Largest Time Gap Largest number of gaps in a file Files with >10 msec gap
1/31/13 1,399 728 54260282 1820
4/9/13 1298 263 180 34
4/9/13 1,299 122 95 33
4/9/13 1,299 55 2113 17
4/10/13 1,299 34 50
FINAL 1,346 49 34 26 28

Final Cleaned Seasat Swaths: Approximately one year after the project started, 1,346 raw Seasat swaths were cleaned and ready to be processed into SAR image products.

Written by Tom Logan, July 2013

seasat_equation_clock_drift

Seasat – Technical Challenges – 3. Decoded Data Analysis

With the Seasat archives decoded into range line format along with an auxiliary header file full of metadata, the next step is to focus the data into synthetic aperture radar (SAR) imagery. Focusing is the transformation of raw signal data into a spatial image. Unfortunately, pervasive bit errors, data drop outs, partial lines, discontinuities and many other irregularities were still present in the decoded data.

3.1 Important Metadata Fields

In order for the decoded SAR data to be focused properly, the satellite position at the time of data collection must be known. The position and velocity of the satellite are derived from the timestamp in each decoded data segment, making it imperative that the timestamps are correct in each of the decoded data frames.

Slant range is the line of sight distance from the satellite to the ground. This distance must be known for focusing reasons and for geolocation purposes. As the satellite distance from the ground changes during an orbit, the change is quantified using the delay-to-digitization field. During focusing, the slant range to the first pixel is calculated using these quantified values. More specifically, the slant range to the first pixel (srf) is determined using the delay to digitization (delay), the pulse repetition frequency (PRF) and the speed of light (c):

seasat_equation_clock_drift

It turns out that the clock drift is also an important metadata field. Clock drift records the timing error of the spacecraft clock. Although it is not known how this field was originally created, upon adding this offset to the day of year and millisecond of day more accurate geolocations were obtained in the focused Seasat products.

Finally, although not vital to the processing of images, the station code provides information about the where the data was collected and may be useful for future analysis of the removal of systematic errors.

3.2 Bit Errors

It is assumed that the vast majority of the problems in the original data are due to bit errors resulting from the long dormancy of the raw data on magnetic tapes. The plots in section 2.1 showed typical examples of the extreme problems introduced by these errors, as do the following time plots.

Time Plot: Very regular errors occur in much of the data, almost certainly some of which are due strictly to bit errors. Note that this plot should show a slope, but the many errors make it look flat instead.
Time Plot: This plot shows a typical occurrence in the Seasat raw data: Some areas of the data are completely fraught with random errors; other areas are fairly “calm” in comparison.

3.3 Systematic Errors in Timing

Beyond the bit errors, other, more systematic errors affect the Seasat timing fields. These include box patterns, stair steps and data dropouts.

To top off the problems with the time fields, discontinuities occur on a regular basis in these files. Some files have none; some have hundreds. Some discontinuities are small — only a few lines. Other discontinuities are very large — hundreds to thousands of lines. Focusing these data required identifying and dealing with discontinuities.

Box errors
Box Errors: Regular patterns of errors occurred in many datatakes. This is most likely the result of faulty hardware either on the platform or ground station. Note this plot also shows the cleaned times in green. See the next section for details on how this was accomplished.
Stair Steps
Stair Steps: This plot shows a small section of time data from lines 218500-218700 of one Seasat header file. Readily obvious are some random bit errors and the fairly typical “stair step” error. It is assumed that the stair steps are the result of a sticking clock either on the satellite or in the receiving hardware.
Data Dropouts
Forward Time Discontinuity
Backward Time Discontinuity
Double Discontinuity
Time Discontinuity

Aside: Initial Data Quality Assessment

Of the 1,470 original decoded data swaths

  • Datasets with Time Gaps (>5 msec): 728
  • Largest Time Gap: 54260282
  • Largest Number of Gaps in a Single File:1,820
  • Number of files with stair steps: 295
  • Largest percentage of valid repeated times: 63%
  • Number of files with more than one partial line: 1,170
  • Largest percentage of partial lines: 42%
  • Number of files with bad frame numbers: 1,470
  • Largest percentage of bad frame numbers: 17%

Sea Ice MEaSUREs – How to Cite

Citing Sea Ice MEaSUREs Datasets

Cite datasets in publications such as journal papers, articles, presentations, posters, and websites. Each Sea Ice MEaSUREs dataset has an assigned DOI. Please send copies of, or links to, published works citing data, imagery, or tools accessed through ASF to uso@asf.alaska.edu with “New Publication” on subject line.

Format Example
[Last name, first initial of principal investigator if appropriate*], RADARSAT-1 data [year of data acquisition] (CSA). Dataset: [name of dataset]. Retrieved from ASF DAAC [add URL if print publication: asf.alaska.edu], [day month year of data access]. DOI: [doi].

*Kwok, R., is the principal investigator on all three Sea Ice MEaSUREs datasets.
Kwok, R., RADARSAT-1 data 1997 (CSA). Dataset: Lagrangian Sea-Ice Kinematics. Retrieved from ASF DAAC 7 December 2014. DOI: 10.5067/SSMPINYI15UU.

Citing Sea Ice MEaSUREs Imagery

Include appropriate credit with each image shown in publications such as journal papers, articles, presentations, posters, and websites.

Type Format Example
Imagery, primary or altered © CSA data [year of data acquisition]. © CSA data 2014.
Imagery, derivative © Kwok, R., [year of image creation], contains CSA data [year of data acquisition]. © Kwok, R., 2012, contains CSA data 1997.

Sea Ice MEaSUREs Datasets and DOIs

Datasets DOI
Lagrangian Sea-Ice Kinematics Dataset 10.5067/SSMPINYI15UU
Three-day Gridded Sea-Ice Kinematics Data 10.5067/GWQU7WKQZBO4
Eulerian Sea-Ice Kinematics Dataset 10.5067/JQHSWB5Y45AY
Artist's concept of SMAP observatory and its antenna-beam footprint. Credit: NASA.

SMAP – Instrument

Soil Moisture Passive Active (SMAP) is a remote-sensing observatory with two instruments — a synthetic aperture radar (SAR) and a radiometer — that map soil moisture and determine the freeze or thaw state of the area being mapped….

SMAP – Handbook

“A rare characteristic of the SMAP Project is its emphasis on serving both basic Earth System science as well as applications in operational and practice-oriented communities.”

Contents of Full Handbook

1. Introduction and Background
2. Mission Overview
3. Instrument Design and L1 Data Products
4. Soil Moisture Data Products
5. Value-Added L4_SM Soil Moisture Product
6. Carbon Cycle Data Products
7. Science Data Calibration and Validation
8. NASA SMAP Applications Program
9. SMAP Project Bibliography

SMAP Handbook Excerpts on ASF's Roles

The Alaska Satellite Facility (ASF) is one of four ground stations that support the SMAP mission and one of two NASA DAACs that distribute SMAP data. The SMAP baseline science data products will be generated within the project’s Science Data System and made available publicly through the two NASA-designated Earth-science data centers. The ASF Synthetic Aperture Radar (SAR) Distributed Active Archive Center (DAAC) will provide Level 1 radar products, and the National Snow and Ice Data Center (NSIDC) DAAC will provide all other products. The excerpts below from the SMAP Handbook focus on ASF’s roles.

About ASF
ASF, part of the University of Alaska Fairbanks (UAF) Geophysical Institute, operates the SAR DAAC for NASA. For more than 20 years, ASF has worked in conjunction with the SAR research community and scientists across the globe providing near-real-time and archive data from several key Earth-observing satellites. In support of this user community, ASF offers interactive web resources for data search and download, and creates custom software tools for data interpretation and analysis.

ASF’s DAAC is one of 12 Data Centers supported by NASA and specializes in the processing, archiving, and distribution of SAR data to the global research community. In recent years, the ASF DAAC has moved from a process-on-demand to a download-on-demand data system that provides direct access to over 1 PB of SAR data. The ASF data system, comparable to the EOSDIS Core System, provides ingest, cataloging, archiving, and distribution of ASF DAAC’s complete data holdings. ASF distributes focused and unfocused SAR data products, browse images, and relevant metadata in multiple formats through the Vertex data search portal. 

Ground Data System (GDS)
The primary path for commanding the SMAP observatory and returning science and engineering data is through three northern-hemisphere tracking stations and one southern-hemisphere station in Antarctica. Data return at the northern-hemisphere stations is via 11.3-m antennas located at Wallops, Virginia (WGS), Fairbanks, Alaska (ASF), and Svalbard Island, Norway (SGS). Data return at the southern-hemisphere station is via the 10-m antenna at McMurdo Station, Antarctica (MGS). The table below gives characteristics of the four stations and average contact statistics from the science orbit. Because SMAP is in a near-polar orbit, the higher latitude stations have more frequent contact opportunities.

Ground Station Antenna Latitude Average # of Contacts per day* Average Coverage Minutes/day*
Svalbard (SGS) Norway 11.3 m 78.2ºN 10.3 88.3
Fairbanks (ASF) Alaska 11.3 m 64.9ºN 6.8 53.7
Wallops (WGS) Virginia 11.3 m 37.9ºN 3.3 25.8
McMurdo (MGS) Antarctica 10.0 m 77.8ºS 10.4 90.7

ASF DAAC Support of NASA Missions
The ASF DAAC provides support for NASA and NASA-partner missions assigned to it by the Earth Science Data and Information System (ESDIS) Project. The ASF DAAC has extensive experience managing diverse airborne and spaceborne mission data, working with various file formats, and assisting user communities to further the use of SAR data.

These efforts are facilitated, in part, by ASF Scientists and Data Managers, who interact with mission teams, provide subject matter expertise, inform data and metadata formats, evaluate data structure and quality, and address data support needs. A key project component at ASF is the core product team, which provides integration of new datasets into the ASF data system and ensures efficient coordination and support of each mission. The team members have mission-specific expertise and consist of the following personnel:

  • The Project Manager is the team leader who oversees mission activities at ASF and coordinates with external groups.
  • The Product Owner is a primary product stakeholder and oversees ingest, archive, documentation, and distribution of data products as well as managing interactions with mission and ASF scientists and other stakeholders.
  • The User Services Representative (uso@asf.alaska.edu) supports data users with products and software tools and communicates user feedback or suggestions for improvement to the Project Manager and Product Owner. 
  • Software Engineers design, develop, and maintain software for the acquisition, processing, archiving, and distribution of satellite and aerial remote sensing data.
  • Software Quality Assurance Technicians provide software and web-based-application testing prior to delivery to the production data system to ensure integrity, quality, and overall proper functionality through testing methods to uncover program defects, which in turn are reported to software engineers.
  • The Technical Science Writer composes and edits a variety of ASF materials, from newsletter articles to technical documentation.

The core product team’s responsibilities for data management include:

  • Ingesting, cataloging, archiving, and distributing data
  • Providing guidance on file formats and integration of new file formats into the ASF data system
  • Describing data products and producing user manuals and guide documents
  • Creating metadata and exporting it to CMR and GCMD (Global Change Master Directory)
  • Ensuring accurate metrics are reported to EMS (ESDIS Metrics System)
  • Designing, developing, and deploying specialized data portals that allow online access to data products and information 
  • Creating software tools for data interpretation and analysis
  • Assisting users with the selection and usage of data

ASF also supports NASA and partner missions through the operation of a ground station with two 11-m antennas, providing complete services, including data downlinking, commanding, and range/Doppler tracking. ASF is part of the NASA Near Earth Network (NEN) supporting a variety of low-Earth-orbit spacecraft.

ASF DAAC Data Systems
The ASF DAAC operates a custom data system designed, implemented, and supported by DAAC personnel. During its evolution, the ASF data system has moved from using primarily custom software on capital equipment to commodity hardware and commercial off-the-shelf (COTS) software and hardware solutions. This has greatly lowered development and maintenance costs for the data system, while simultaneously providing a higher level of performance. The ASF DAAC data system provides the following capabilities:

Data Ingest

  • Automated data ingest occurs from the ASF ground station as well as external data providers in a variety of media and formats.
  • Ingested data are pre-processed when necessary, providing browse or derivative products.

Data Archive

  • The central ASF data system archive is provided by a Data Direct Networks gridscaler storage system.
  • This system provides direct access to over 1 PB of processed data as well as the capability for automated backups to an offsite location.
  • Raw data are held in a robotic silo for access by the processing system. ASF maintains a backup in an external location in case of silo failure.

Data Distribution

  • ASF provides direct http access to DAAC data products and utilizes NASA’s User Registration System (URS) for user authentication. 
  • NASA data are provided to public users with no restrictions. Partner data are provided to NASA-approved users through URS for authentication and ASF’s internal database for access control.
  • The data system provides web-based access to the archive through Vertex. Vertex supports the data pool with direct download of processed data.
  • Through custom portals and applications, the DAAC provides additional services such as mosaic subsetting, mosaicking, and time-series analysis.

Data Support 

  • ASF DAAC exports relevant metadata to NASA’s ECHO system.
  • ASF DAAC exports ingest, archive, and download metrics to NASA’s EMS system.
  • ASF DAAC assists users with data discovery and usage, maintains product documentation and use guides, and supports feedback between the ASF user community and the core product teams. 

SMAP at ASF DAAC
ASF provides a variety of services, software tools, and user support to address the needs of the SMAP user community. The ASF core project team will leverage on-going collaborations with the SMAP Project to identify and prioritize SMAP user community needs, which in turn will inform development and implementation of data support and value-adding services for the mission. The SMAP website at ASF will serve as an interactive data portal, providing users with relevant documentation, custom tools and services, and ancillary data and resources.

Post-Launch SMAP Data
ASF will ingest, distribute, archive, and support postlaunch Level 1 radar products for the SMAP mission. ASF will receive the Level 1 radar products from the SMAP Science Data System at the Jet Propulsion Laboratory (JPL) in Pasadena, California.

Non-SMAP Data of Interest to SMAP
ASF will cross-link from the SMAP website to data collections that complement SMAP data and are of interest to the user community. Some of these collections are distributed by ASF, including the following:

  • Airborne Microwave Observatory of Subcanopy and Subsurface (AirMOSS) data products
  • Jet Propulsion Laboratory Uninhabited Aerial Vehicle SAR (UAVSAR) data products
  • Making Earth System Data Records for Use in Research Environments Inundated Wetlands (MEaSUREs) data products
  • Advanced Land Observing Satellite-Phased Array L-band SAR (ALOS PALSAR)
  • Japanese Earth Resources Satellite-1 (JERS-1) image data and mosaics

Wetlands MEaSUREs – Monitoring Instruments

Spaceborne microwave remote sensing offers effective tools for characterizing wetlands, as microwave sensors are particularly sensitive to surface water and to vegetation structure, allowing monitoring of large, inaccessible areas on a temporal basis regardless of atmospheric conditions or solar illumination….