cropland_data

SMAP – Data and Imagery

SMAP Data Products

Instrument events timetable

The SMAP baseline science-data products will be publicly available through two NASA-designated data centers, the Alaska Satellite Facility (ASF) and the National Snow and Ice Data Center (NSIDC) DAAC. ASF will distribute the Level 1 radar products and NSIDC will distribute the radiometer and Level 2-4 products. All products will conform to the HDF-5 standard.

SMAP synthetic aperture radar (SAR) data have a spatial resolution of 1-3 km over the outer 70% of the swath (L1C_S0_HiRes product). The low-resolution radar data are ‘slices’ of resolution approximately 5 km x 30 km (L1B_S0_LoRes product). The radiometer data have a spatial resolution (IFOV) of 39 km x 47 km, nominally referred to as 40-km resolution (L1B_TB product). The L1C_TB data products are resampled TB data on a 36-km Earth grid and will have spatial resolution ~10% greater than the L1B_TB data (depending on the resampling method).

SMAP science-data products will be generated on the Science-Data System production system using science software developed on the SDS Testbed. The science software is based on algorithms for each product described in the Algorithm Theoretical Basis Documents (ATBDs). 

(Beta Level 1 radar products now available for download)

Short Name Description Gridding (km) Latency* ATBD** Product Spec Doc Source
L1A_Radar Raw radar data in time order (half orbit) - 12 hrs n/a View ASF
L1A_Radiometer Radiometer raw data in time order - 12 hrs n/a NSIDC
L1B_S0_LoRes Low-resolution radar σ ο in time order (half orbit)
Backscatter analysis document
5x30 12 hrs View View ASF
L1B_TB Radiometer TB in time order 36x47 12 hrs View NSIDC
L1C_S0_HiRes High-resolution radar σ ο (half orbit gridded)
Backscatter analysis document
1
(1-3)***
12 hrs View View ASF
L1C_TB Radiometer T B (half orbit, gridded) 36 12 hrs View NSIDC
L2_SM_A Soil moisture (radar, half orbit) 3 24 hrs View NSIDC
L2_SM_P Soil moisture (radiometer, half orbit) 36 24 hrs View NSIDC
L2_SM_A/P Soil moisture (radar/radiometer, half orbit) 9 24 hrs View NSIDC
L3_F/T_A Freeze-thaw state (radar, daily composite) 3 50 hrs View NSIDC
L3_SM_A Soil moisture (radar, daily composite) 3 50 hrs View NSIDC
L3_SM_P Soil moisture (radiometer, daily composite) 36 50 hrs View NSIDC
L3_SM_A/P Soil moisture (radar/radiometer, daily composite) 9 50 hrs View NSIDC
L4_SM Soil moisture (surface and root zone) 9 7 days View NSIDC
L4_C Carbon net ecosystem exchange (NEE) 9 14 days View NSIDC

*Mean latency under normal operating conditions (defined as time from data acquisition by the observatory to availability to the public data archive).

**Algorithm Theoretical Basis Documents (ATBDs) provide the physical and mathematical descriptions of the algorithms used in the generation of science-data products. The ATBDs include a description of variance and uncertainty estimates and considerations of calibration and validation, exception control, and diagnostics. Internal and external data flows are also described.

***Over outer 70% of swath

Ancillary Data Reports

The SMAP Ancillary Data Reports below hold application-related descriptions of datasets used with science-algorithm software in generating SMAP science-data products, as well as links to EASE grid information relevant to SMAP products.

These reports may undergo additional updates as new ancillary datasets or processing methods become available.

Antarctic Coastline

International Polar Year – GIIPSY Documents, Publications, and Mission Plans

Documents

Science Data Requirements and Mission Planning

The following files summarize a community assessment of GIIPSY satellite data requirements. Detailed requirements and justifications are discussed in the GIIPSY Science Requirements Document. A consolidated and numbered set of science themes are listed in GIIPSY Thematic Objectives. These were presented and tentatively adopted by the Space Task Group. A spread sheet linking GIIPSY thematic objectives by number to specific observational requirements is included as the GIIPSY Check List. The GIIPSY Check List also refers to image and digital maps listed below.

GIIPSY Science Requirements Document
GIIPSY Thematic Objectives
GIIPSY Check List

Image Maps of Coverage

Antarctica and Greenland Ice Sheets

Arctic and Antarctic Sea Ice

Glaciers

Permafrost

Mission Plans

Sea Ice MEaSUREs – Common Questions and Applications

What is Sea Ice?

The simple answer is that sea ice forms in the sea.
That's when it starts to get interesting.

Drift ice is not frozen to anything stationary, such as a shore, the ocean bottom, or grounded ice. In the Arctic Ocean, where drift ice can move as much as 10 km a day, winds and currents tend to transport the floating ice in two patterns. The Beaufort Gyre moves in a clockwise rotation in the Beaufort Sea, and the Transpolar Drift Stream (see map), flows from the Arctic Ocean toward the Atlantic. As Arctic sea ice has thinned in recent decades, its drift speed has tended to increase.

A synthetic aperture radar (SAR) image of the Bering Strait reveals ice, land, and water (black) in January. Radar sees through clouds and dark. Russia is on the left; Alaska is on the right. © CSA 2008

Fast ice, sometimes called landfast ice, freezes to the shore, the ocean bottom, an ice wall, or a grounded iceberg. If it rises more than about two meters above sea level, it is called an ice shelf.

First-year sea ice has lasted for no more than one winter, but it is thicker than the thin films of new ice called frazil, grease ice, and nilas. Although first-year ice ejects salt from its ice crystals in the freezing process, it still contains pockets of saltwater. If first-year ice survives to become multiyear ice, the saltwater eventually drains out, and the ice becomes denser. As Arctic sea ice has diminished in recent decades, first-year ice has become more prevalent than multiyear ice.

Sea-ice floes are individual pieces of sea ice 20 m or more across.

Frazil is the start of sea ice — a collection of ice crystals that form a film on the water. As the crystals form, they eject salt into the surrounding water. The saltier the water, the heavier it is, so it sinks. The ocean becomes layered with sea ice on top of cold water, on top of heavier, saltier water. Those layers move in currents around the world, playing a large role in global climate.

Grease ice is the next stage after frazil, when the ice crystals assemble in a syrupy layer.

Grease ice occurs after frazil. © WHOI 2014.

Ice shelves are landfast sea ice (also known as fast ice) that rise more than about two meters above sea level.

Lagrangian sea-ice data products contain monthly measurements of dynamic and kinematic parameters over the Arctic Ocean sea-ice cover. “Lagrangian” refers to a mathematical way to study ice dynamics by noting changes in position and velocity of points over time. The sea-ice analysis in the dataset available here is initialized by laying a 10km-by-10km grid over a set of RADARSAT-1 images of the sea ice during an initial 3-day period of a season. The grid forms cells for which a number of properties can be derived that describe the sea-ice dynamics. The grid points of the sea ice and related cell properties are then tracked throughout the season.

Leads are long, linear areas of open water that develop as sea ice pulls apart, moved by winds or currents. A lead in sea ice can range from a few meters to over a kilometer in width and tens of kilometers long. Leads and polynyas (see below) are important for wildlife. The world’s population of spectacled eiders, for example, spends each winter in the northern Bering Sea, depending on the water in leads and polynyas when most other open water is frozen.

Sea ice melt ponds form in depressions on top of the ice, deepening them. Because ice serves as a reflective surface that absorbs little solar energy, the darker water of melt ponds significantly warms both the ice and the atmosphere, accelerating the melting. 

Sea ice melt ponds accelerate melting. Credit: NASA 2011

Multi-year sea ice has survived at least two summers. Unlike first-year ice, multi-year sea ice is almost free of salt. In recent years, multi-year sea ice has become thinner.

Pack ice is a term sometimes used to refer to particularly closely packed sea ice. It is sometimes used interchangeably with drift ice (see above).

Pack ice off of Baffin Island, Canada. © 2013 Paul Gierszewski.

Pancake ice consists of flat, circular pieces of sea ice that form from frazil and/or grease ice in rough water. They also can form below the surface. The edges of the “pancakes” are characteristically thickened from colliding in winds, waves, or currents in a sort of bumper-car effect.

Polynyas, like leads (see above), are large open areas of water surrounded by sea ice — or by sea ice on one side and land on the other. Polynyas, however, are wide, not long and narrow. Polynyas and leads are important for wildlife.

Sea-ice pressure ridges form when wind and currents press ice against sea ice, causing the sea ice to buckle and/or layer on top of itself.

Rafting takes place when winds or currents push pieces of sea ice on top of each other.

The RADARSAT-1 Geophysical Processor System (RGPS) was designed to analyze the large volumes of SAR imagery required to produce basin-wide fields of sea-ice motion and estimates of sea-ice age and thickness from sequential RADARSAT-1 maps of the Arctic and Southern Oceans. From the RGPS, researchers have been able to obtain routine estimates of the seasonal sea ice produced by deformation of the sea-ice cover.

Applications

Sea ice is a key indicator of environmental change and is also a force to be reckoned with for humans navigating, drilling, or fishing in polar waters. At the northern end of the planet, where the Arctic’s oceans are largely constrained within landmasses — and the impact of climate change is more dramatic than anywhere else — sea ice serves as a particularly significant laboratory for monitoring our changing planet.

The data and imagery available from the Alaska Satellite Facility cover a period from 1995 to 2011. They include more than 11 years of near-uninterrupted, three-day radar snapshots of Arctic and Southern Ocean’s sea ice as it goes through dramatic change. Users may access:

  • A dataset of small-scale kinematics and deformation processed by tracking sea ice on a high-resolution grid. 
  • The original synthetic aperture radar (SAR) images.

Uses

This dataset has already contributed significantly to sea-ice science in:

  • new approaches for modeling the mechanical behavior of sea ice and the validation of these models;
  • characterization of sub-daily ice motion;
  • description of the seasonal and regional variability of sea-ice deformation;
  • validation of ICESat freeboard algorithms; and
  • estimates of sea-ice exchange between the Arctic and Southern Oceans and peripheral seas. 
seasat_data_flow_prep_cleaning

Seasat – Technical Challenges – 4. Data Cleaning (Part 2)

4.3 Prep_Raw.sh

seasat_data_flow_prep_cleaning

After development of each of the software pieces described previously in this section, the entire data cleaning process was driven by the program prep_raw.sh. This procedure was run on all of the swaths that were output from SyncPrep to create the first version of the ASF online Seasat raw data archive (the fixed_ files). Analysis of these results is covered in the next section. What follows here are examples of the prep_raw.sh process and intermediate outputs.

Time Filtering in Stages: Each plot shows range line number versus MSEC metadata value. Top row is before filtering; bottom is after. From left to right: (a) Stage 1 — attempt to fix all time values > 513 from the linear trend. (b) Stage 2 — fix stair steps resulting from sticking clock on satellite. (c) Stage 3 — final linear fix before discontinuity removal.

Time Filtering in Stages: Each plot shows range line number versus MSEC metadata value. Top row is before filtering; bottom is after. From left to right: (a) Stage 1 — attempt to fix all time values > 513 from the linear trend. (b) Stage 2 — fix stair steps resulting from sticking clock on satellite. (c) Stage 3 — final linear fix before discontinuity removal.

Data Set #1

Decoded Signal Data
Time Gap Corrections
Stair Corrections
Time Value Corrections

Data Set #2

Decoded Signal Data
Time Gap Corrections
Stair Corrections
Linear Time Restored
Decoded Time Data from Section 2.1 Examples
Cleaned Time Data from 2.1 Examples

4.4 Results of Prep_Raw.sh

By November 15, 2012, the beta version of prep_raw.sh was delivered to ASF operations. It was run on individual swaths at first, with results spot-checked. Once confidence in the programs increased, all remaining Seasat swaths were processed through this decoding and cleaning software en masse. Overall, from the 1,840 files that SyncPrep created, 1,470 were successfully decoded, and 1,399 of those made it through the prep_raw.sh procedure to create a set of fixed decoded files. The files that failed comprise 242 GB of data, while 3,318 GB of decoded swath data files were created.

Processing Stage #files Size (GB)
Capture 38 2160
SyncPrep 1840 2431
Original Decoded 1470 3585
Fixed Decoded 1399 3318
Good Decoded 1346 3160
Bad Decoded 53 157

Summary of Data Cleaning

93% of data made it through SyncPrep
92% of that data decoded (assume 1.6 expansion)
93% of that data was “fixed”
95% of that data considered “good”
OVERALL: ~80% of SyncPrep’d data is “good”

Reasons for Failures

  • SyncPrep: Not all captured files could be interpreted by SyncPrep
  • Original Decoded: Because the decoder needs to interpret the subcommutated headers, it is more stringent on maintaining a “sync lock” than SyncPrep
  • Fixed Decoded: Some files are so badly mangled that a reasonable time sequence could not be recovered
  • Bad Decoded: Several subcategories of remaining data errors are discussed in section 5

4.5 Addition of fix_start_times

During analysis of the fixed metadata files, it was discovered that bad times occurred at the beginning of many files. This problem was not a big surprise; the nature of the sync code search is such that many errors occur in places where the sync codes cannot be found. This is why the files were broken in the first place. So, it is expected that the beginning of a lot of the swath files will have bad metadata, which means bad times. When bad times are linearly trended, the results are unpredictable.

Bad Start Time Examples

To fix this problem, yet another level of filtering was added to the processing flow – this time a post-filter to fix the start times. The code fix_start_times replaces the first 5,000 times in a file with the linear trend resulting from the next 10,000 lines in the file. This code was added as a post-processing step to follow prep_raw.sh and run on all of the decoded swath files.

Final Processing Flow: With the addition of the fix_start_times code, the processing flow for data decode and cleaning is finally completed.

Final Processing Flow: With the addition of the fix_start_times code, the processing flow for data decode and cleaning is finally completed.

Fixed Start Time Examples

Written by Tom Logan, July 2013

Bits Per Sample: This value should always be 5, as the parameter never changed during the entire Seasat mission.

Seasat – Technical Challenges – 2. Decoder Development

Starting in the summer of 2012, ASF undertook the significant challenge of developing a Seasat telemetry decoder in order to create raw data files suitable for focusing by a synthetic aperture radar (SAR) correlator. In this case, that means processible by ROI, the Repeat Orbit Interferometry package developed at Jet Propulsion Laboratory. In addition to creating the range lines out of minor frames, the decoder must interpret the 18 fields in the headers to create a metadata file describing the state of the satellite when the data was collected.

The main challenges in decoding the raw telemetry were:

  1. Overcoming bit error problems
  2. Properly forming major lines from a variable number of minor frames
  3. Maintaining sync lock
  4. Discovering sentinels marking data collection boundaries

2.1 Problems with Bit Fields

All 10 of the bit fields proved to be unreliable, and, thus, with the exception of the fill flag, they are ignored by all of the software developed during this project. This section describes the ways in which the bit fields are unreliable.

Each Seasat minor frame contains 8 bits to record time and status. These bits encode 18 metadata fields, subcommutated in the first 10 minor frames of each range line. There are 10 bit fields, four fields that should be constant for a data take, two fields that should change rarely and two fields that should change steadily. Unfortunately, due to the high bit error rate (BER) of the telemetry data, even fields that should be constant show high variability. The following plots, showing decoded metadata values plotted over 800,000 range lines, drive home the enormity of the bit error problems in these data.

Bits per Sample

Bits Per Sample: This value should always be 5, as the parameter never changed during the entire Seasat mission.

PRF rate code

PRF Rate Code: The PRF rate code should be a 4 for the entire mission, since this satellite parameter never changed.

Last Digit of Year

Last Digit of Year: The Seasat mission occurred entirely in 1978, so the last digit of the year should always be 8.

Station Code

Station Code: The Station Code should be a constant for any given datatake. 5: Fairbanks, Alaska; 6: Goldstone, Calif.; 7: Merrit Island, Fla.; 9: Oak Hanger, United Kingdom; 10: Shoe Cove, Newfoundland.

Day of Year

Day of Year: For a given datatake, the day of year should change at most once, since any single datatake cannot exceed even an hour in duration, much less an entire day.

Delay to Digitization

Delay to Digitization: The delay to digitization parameterizes the time between emission of a pulse from the satellite and recording of return echoes. Used to calculate the slant range to the first pixel, the delay should change only a handful of times in any given raw data file based upon changes in orbital altitude.

Clock Drift

Clock Drift: The spacecraft clock drift records the timing error of the spacecraft clock. This should be a smoothly changing field, generally in the 2,000- to 3,000-millisecond range. It is not known how this field was originally created, only that it is vital in getting reasonable geolocations for processed imagery.

MSEC

MSEC: This field records the millisecond of the day when the data were acquired and should, therefore, be a linearly increasing field with an exact slope of 1/PRF.

Given all of the fallout from these truly bizarre plots, it is no surprise that attempts to use the fill flag quickly proved difficult; the bit errors are so pervasive that the field is unreliable.

2.2 Determining Minor Frame Numbers

Several oddities in the raw data are exacerbated by the high BER. First, the data are organized into 1,180-bit minor frames. This means that they are each 147.5 bytes long. Although the .5 byte offset was easy to deal with, it turns out that sync codes may actually appear at 147, 147.5, or 148 bytes from each other at seemingly random places in the raw data file – a topic addressed in section 2.3

Moreover, a variable number of minor frames need to be combined to create a single range line. Some lines contain 59 minor frames and some contain 60. Considering that the frame number in the minor frames is only 7 bits, and no major frame numbers exist in the telemetry, the “simple” task of finding the start of each major line was at times quite difficult. Synchronization codes can be either byte-aligned or non-byte-aligned, and partial lines occur on a regular basis. As a result, the minor frame numbers eventually had to be determined by context.

The current frame number is determined using three previous minor frame numbers and the next frame number — along with a handful of heuristics. For example, if a gap is found in consecutive minor frame numbers, the following rules are applied:

  • If the next frame is 1 and last was either 59 or 60, assume this is frame zero
  • Else if (next_frame-last_frame)==2, put this frame in sequence
  • Else if last two frames are in sequence, try to put this one in sequence – If the last frame < 59, put this in sequence – If the last frame was 59 and the last line was length 60, then this HAS to be frame zero since we never have two length 60 lines in a row.
  • Else if last2 and last3 frames are in sequence and last frame is 0, then set this frame to 1
  • If we got to here, then we did not fix the error!

Even beyond these rules, additional checks for a bad frame number 0 and major frames that spuriously showed more than 60 minor frames still had to be performed.

Aside: Bit Errors in Frame Numbers

Random bit errors in the middle of a line: In these decoded minor frame numbers, we see that lines 1, 2, 4 and 5 have no frame number errors; they are in sequence starting from 1 and going up to 59 or 60 by ones. Meanwhile, line 3 has 24 frame numbers in a row that are in error.

Partial lines: The first line is missing minor frames 5-9; the second line is complete; the third line has repeated minor frames 15-19; and the fourth line is the decoder’s attempt to enforce the fact that at most 60 minor frames form a range line. The fifth line is missing minor frames 23-26; the sixth line is complete; the seventh line has repeated minor frames 32-36; and, again, the eighth line is the decoder’s attempt to enforce the fact that at most 60 minor frames form a range line. 

Multiple lines of bit errors: This example shows how bad random bit errors can be, even if no minor frames are actually missing. Incredibly, in five lines, 122 minor frames are in error out of 298 total, giving a 40 percent error rate for these frame numbers. Perhaps even more incredibly, the ASF Seasat decoder managed to fix all of these frame numbers.

  • Original non-fixed frame numbers: 

  • Frame numbers fixed by the ASF Seasat decoder:

2.3 Maintaining Sync Lock

One very important aspect of decoding telemetry data is maintaining a sync lock: The decoding program must be able to find the synchronization codes that occur at the beginning of each minor frame.

Early in the development of the decoder, it was determined that the sync codes are just as susceptible to bit errors as the rest of the data. Initially, finding sync codes required a considerable amount of searching in the file, with the hope that no false positives would be encountered. After much development and testing, it was determined that in order to maintain sync lock, some number of bit errors had to be allowed in the sync code. Therefore, the code was configured to allow 7 bit errors per sync code out of 24 bits. Values less than this needlessly split datatakes (single passes of data over a given ground station). Values greater than this showed too many “false positive” matches for sync codes.

As a result of this extensive analysis, a pattern was determined in the location of sync codes. That is, a byte-aligned, 147.5-byte frame followed by a non-byte-aligned, 147.5-byte frame, repeated 14,217 times, followed by a single instance of a 147-byte frame. In code form:

Once this pattern was established, most problems with locating sync codes were abated.

2.4 Data Sentinel Values – Breaking Datatakes

The next problem involved bad sections of data that defied attempts to match frame numbers. The only solution is to break the datatake into multiple pieces, closing the current output file when problems arise, and creating a new output file when sync is regained. This is much like what SyncPrep does, except that the ASF decoder has to be more stringent in its rules for maintaining sync since it must be able to properly build range lines in addition to just finding sync codes.

In addition to losing sync lock as a result of BER, two additional cases arose that will break a datatake into segments: either 60 occurrences of the fill flag in a row, or the repeated occurrence of frame number 127. The fill flag is a valid field but is so unreliable it can only be trusted to be correct after many consecutive hits. The frame number 127 showed to be a sentinel for no data; it occurred thousands of times in areas where no valid SAR data was being collected. Either of these happenings will also cause the ASF Seasat decoder to close the current output file and create a new one.

2.5 Results of Decoding

seasat_decoder:

  • Decode raw signal data into unpackaged byte signal data (.dat file)
    • 13680 unsigned bytes of signal data per line
    • File size is aways lines * 13680 bytes in length
  • Decodes all headers to ASCII (.hdr file)
    • 20 columns of integer numbers per line
    • One line entry per line of decoded signal data
  • Additional Features:
    • Allows both byte aligned and anon-byte aligned minor frames
    • Deals with variable length lines, partial lines
    • Fixes frame numbers from context if possible
    • Creates one or more output files per input based on sentinels
    • Assembles headers spread across 10 minor frames

In spite of all of the challenges and problems in the raw data, the ASF Seasat decoder is able to decode raw telemetry SAR data. Using five frame numbers in sequence and a handful of heuristics, telemetry data is decoded into byte-aligned, 8-bit samples. Concurrently, all of the metadata stored in the headers is decoded and placed in an external file.

The current strategy tried to err on the side of only allowing valid SAR data to be decoded. Still, 7-bit errors had to be allowed in a sync code match to even get through the raw data. In addition, the decoded header information is simply not reliable. For example, early in development, the ASF Seasat decoder broke one 7-GB chunk of raw data into 24 segments of decoded data, dumping a header at the beginning and ending of each segment. Analysis of the decoded times in these headers showed that of the 48 dumped, 3 were completely zero and an additional 12 were in error. In other words, the decoded times did not make sense in context with the surrounding time values.

Thus, even after completing the decoder development with bit error tolerance, frame number heuristics, proper sync code detection, and known sentinel values for good data boundaries, the decoded Seasat archives were still nearly unusable in any reliable fashion.

Processing Stage #Files Size (GB)
Capture 38 2610
SyncPrep 1840 2431
Original Decoded 1470 3585

Initial Data Recovery: 93 percent of the data captured from tape made it through SyncPrep; Approximately 92 percent of that data was decoded (assuming a 1.6-expansion factor).

Aside: ASF Tape Archive File Names

When the tapes were captured onto disk, files were named based upon tape number and section of tape read. For example, the first part of tape1 was initially named SEASAT_tape1_01Kto287K.

This file was run through SyncPrep, which created multiple subfiles based upon its ability to maintain a sync lock, sometimes creating over 100 such numbered files, e.g. SEASAT_tape1_01Kto287K.000 to SEASAT_tape1_280Kto668K.020

Next, the files go through the ASF Seasat decoder, gaining yet another subfile number, but the prefix “SEASAT_” is removed. Note that this stage creates a file pair of {.dat, .hdr}, e.g. tape1_01Kto287K.018_000, tape1_01Kto287K.018_001, and tape1_01Kto287K.018_002 file pairs were all created from a single decode of SEASAT_tape1_280Kto668K.018.

Thus, for a single captured file, SyncPrep could make tens to a few hundred data segments, while the ASF Seasat decoder could break each of these files into even more sub-segments.

Written by Tom Logan, July 2013

ROI Configuration File Creation Program: Many command line options are offered by create_roi_in, including the ability to process any amount of signal data or the exact amount of signal data needed to create a single 100-km2 image framed by European Space Agency (ESA) framing standards.

Seasat – Technical Challenges – 9. From Swaths to Products

At this stage in the development of the ASF Seasat Processing System (ASPS): 1,346 cleaned raw signal swaths were created; ROI was modified to handle Seasat offset video format; New state vectors were selected for use over two-line elements (TLE’s); Caltones were filtered from the range power spectra; Data window position files were created…

Seasat - Station Code - Raw Headers

Seasat – Technical Challenges – 4. Data Cleaning (Part 1)

In order to create a synthetic aperture for a radar system, one must combine many returns over time. For Seasat, a typical azimuth reference function — the number of returns combined into a single focused range line — is 5,600 samples. Each of these samples is actually a range line of radar echoes from the ground. Properly combining all of these lines requires knowing precisely when a range line was received by the satellite.

In practice, SAR systems transmit pulses of energy equally spaced in time. This time is set by the pulse repetition frequency (PRF); for Seasat, 1,647 pulses are transmitted every second. Alternatively, it can be said a pulse is transmitted every 0.607165 milliseconds, an interval commonly referred to as the pulse repetition interval or PRI. Without this constant time between pulses, the SAR algorithm would break down and data would not be focused to imagery.

Many errors existed in the Seasat raw data decoded at ASF. As a result, multiple levels of filtering were required to deal with issues present in the raw telemetry data, particularly with time values. Only after this filtering were the raw SAR data processible to images.

4.1 Median Filtering and Linear Regression

The first attempt at cleaning the data involved a simple one-pass filter of the pertinent metadata parameters. The following seven parameters were median filtered to pull out the most commonly occurring value: station code, day of year, clock drift, delay to digitization, least significant digit of year, bits per sample and PRF rate code. A linear regression was used to clean the MSEC of Day metadata field. This logic is encapsulated in the program fix_headers, which is discussed in more detail in “Final Form of Fix_Headers.”

Implementing the median filter was straightforward:

  1. Read through the header file and maintain histograms of the relevant parameters.
  2. Use the local median value to replace the decoded value and create a cleaned header file. Here “local” refers to the 400 values preceding the value to be replaced.

This scheme works well for cleaning the constant and rarely changing fields. It also seems to work quite well for the smoothly changing clock drift field. At the end of this section are the examples from “Problems with Bit Fields,” along with the corresponding median-filtered versions of the same metadata parameters. In each case, the median filter created clean usable metadata files.

Performing the linear regression on the MSEC of Day field was also straightforward. Unfortunately, the results were far from expected or optimal. The sheer volume of bit errors combined with discontinuities and timing dropouts made the line slopes and offsets highly variable inside a single swath. These issues will be examined in the next section.

Table of Filtered Parameters

Parameter Filter Applied Value
station_code Median Constant per datatake
day_of_year Median Varies
clock_drift Median Varies
delay_to_digitization Median Varies
least_significant_digit_of_year Median 8
bit_per_sample Median 5
prf_rate_code Median 4
msec_of_day Linear Regression Varies

Data Cleaning Examples

Station Code
RAW
NEW
Day of Year
RAW
NEW
Last Digit of the Year
RAW
NEW
Clock Drift
RAW
NEW
Bits Per Sample
RAW
NEW
PRF Rate Code
RAW
NEW
Delay to Digitzation
RAW
NEW

4.2 Time Cleaning

Creating a SAR image requires combining many radar returns over time. This requires that very accurate times are known for every SAR sample recorded. In the decoded Seasat data, the sheer volume of bit errors, combined with discontinuities and timing dropouts, resulted in highly variable times inside a single swath.

4.2.1 Restricting the Time Slope

One of the big problems with applying a simple linear regression to the Seasat timing information was that the local slope often changed drastically from one section of a file to another based upon bit errors, stair steps and discontinuities. Since a pulse is transmitted every 0.607165 milliseconds, it seemed that the easiest way to clean all of the MSEC times would be to simply find the fixed offset for a given swath file and then apply the known time slope to generate new time values for a cleaned header file.

This restricted slope regression was implemented when it became obvious that a simple linear regression was failing. By restricting the time slope of a file to be near the 0.607-msec/line known value, it was assumed that timing issues other than discontinuities could be removed. The discontinuities would still have to be found and fixed separately in order for the SAR focusing algorithm to work properly. Otherwise, the precise time of each line would not be known.

4.2.2 Removing Bit Errors from Times

fix_time

  • Crude time filtering, trying to fix all values that are > 513 from local linear trend:
    • Bit fixes – replace values powers of 2 from trend
    • Fill fixes – fill gaps in constant  consecutive values
    • Linear fixes – replace values with linear trend
  • Reads and writes a file of headers

Even with linear regressions and time slope limitations, times still were not being brought into reasonable ranges. Too many values were in error in some files, and a suitable linear trend could not be obtained. So, another layer of time cleaning was added as a pre-filter to the final linear regression done in fix_headers. The program fix_time was initially created just to look for bit errors, but was later expanded to incorporate each of three different filters at the gross level (i.e. only values > 513 from a local linear trend are changed):

  1. If the value is an exact power of 2 off from the local linear trend, then add that power of 2 into the value. This fix attempts to first change values that are wrong simply because of bit errors. The idea is that this is a common known error type and should be assumed as the first cause.
  2. Else if the value is between two values that are the same, make it the same as its neighbors. This fix takes advantage of the stair steps found in the timing fields. It was only added in conjunction with the fix_stairs program discussed below. The idea is to take advantage of the fact that the stair steps are easily corrected using the known satellite PRI.
  3. Else just replace the value with the local linear trend. At this point, it is better to bring the points close to the line than to leave them with very large errors.

4.2.3 Removing Stair Steps from Times

fix_stairs

  • Fix for sticky time field – Turns “stairs” into “lines” by replacing repeated time values with linear approximation for better linear trend
  • Reads and writes a file of headers

For yet another pre-filter, it was determined that the stair steps time anomaly should be removed before fitting points to a final linear trend. This task is relatively straightforward: If several time values in a row are the same, replace them with values that fit the known time slope of the satellite. The program fix_stairs was developed to deal with this problem.

Seasat - Extreme Stair Step: raw0.headers (blue) are the original unfiltered MSEC time values; newest.headers (red) shows the result of fixing the “stairs” that result from the sticky clock, presumably an artifact of the Seasat hardware, not the result of bit rot, transcription or other errors

Extreme Stair Step: raw0.headers (blue) are the original unfiltered MSEC time values; newest.headers (red) shows the result of fixing the “stairs” that result from the sticky clock, presumably an artifact of the Seasat hardware, not the result of bit rot, transcription or other errors

4.2.4 Final Form of fix_headers

fix_headers

  • Miscellaneous header cleaning using median filters:
    • Station Code
    • Least Significant Digit of Year
    • Day of Year
    • Clock Drift
    • Bits Per Sample
    • PRF Rate Code
    • Delay to Digitization
  • Time Discontinuity Location and Additional Filtering
    • Replace all values > 2 from linear trend with linear trend
    • Locates discontinuities in time, making an annotated file for later use.
      • 5 bad values with same offset from trend identity a discontinuity
      • +1 discontinuity is forward in time and can be fixed
      • if > 4000, too large – cannot be fixed
      • otherwise, slide time to fit discontinuity
      • if offset > 5 time values, save this discontinuity in a file
    • -1 discontinuity is backwards in time and cannot be fixed
  • Reads and writes a file of headers

Although it started out as the main cleaning program, fix_headers is currently the final link in the cleaning process. Metadata going through fix_headers has already been partially fixed by reducing bit errors, removing stair steps, and bringing all other values that show very large offsets into a rough linear fit. So in addition to performing median filtering on important metadata fields (see section 4.1), this program performs the final linear fit on the time data.

Initially, a regression is performed on the first window of 400 points and used to fix the first 200 time values of that window. Any values that are more than 5 msec from their predecessors are replaced by the linear fit. After this, a new fit is calculated every window/2 samples, but never within 100 samples of an actual discontinuity. The final task for fix_headers was to locate the rough locations of all real discontinuities that occur in the files. At least, it was designed to only find real discontinuities – those being the final problem hindering the placement of reasonable linear times in the swath files.

Identifying discontinuities was challenging. Much trial and error resulted in a code that worked for nearly all cases and was able to be configured to work for the other cases as well. The basic idea is that if a gap is found in the data, and if after the gap no other gaps occur within 5 values, then it is possible that a discontinuity exists. If so, the program determines the number of lines that would have to be missing to create such a gap and records the location and size in an external discontinuity file. Note that only forward discontinuities can be fixed in this manner and only discontinuities less than a certain size. In practice, the procedure attempts to locate gaps of up to 4,000 lines, discarding any datasets that show gaps larger than this.

Seasat - Extreme Discontinuity: This decoded signal data shows a 5.3-hour gap in time. This cannot be fixed.

Extreme DiscontinuityThis decoded signal data shows a 5.3-hour gap in time. This cannot be fixed.

In the end, all of the gaps in the data were identified, and, there is high confidence that any such discontinuities found are real and not just the result of bit errors or other problems. Unfortunately, this method was not able to pinpoint the start of problems, only that they existed, as shown in the following set of graphs.

Decoded Signal Data with No Y-range Clipping
Decoded Signal Data with Y-range Clipping
Linear Trend of Decoded Signal Data
Comparison of Decoded Signal Data and Linear Fit
False Discontinuity #1
False Discontinuity #2

4.2.5 Removing Discontinuities

dis_search

  • fix all time discontinuities in the raw swath files
    • for each entry in previously generated discontinuity file:
      • search backwards from discontinuity looking for points that don’t fit new trend line
      • when 20 consecutive values that are ont within 1.5 of new line are found, you have found start of discontinuity
      • For length of discontinuity
        • Repeat header line in .hdr file (fixing the time only)
        • Fill .dat file with random values
    • Reads discontinuity file, original .dat and .hdr file, and cleaned .hdr file.  Creates final cleaned .dat and .hdr file ready for processing

The first task in removing the discontinuities is locating them. The rough area of each real discontinuity can be found using the fix_headers code as described in the previous section.

Finding the exact start and length of each discontinuity still remains to be done. This search and the act of filling each gap thus discovered is performed by the program dis_search. The discontinuity search is performed backward, with 3000 lines after each discontinuity area being cached and then searched for a jump down in the time value to the previous line. These locations were marked as the actual start of the discontinuity. The gap in the raw data between the time before the discontinuity and the time after must then be filled in. Random values were used for fill, these being the best way to not impact the usefulness of the real SAR data.

Discontinuity Fills: Each plot shows range line number versus MSEC metadata value. Original decoded metadata (left) is spotty and contains an obvious time discontinuity. After the discontinuity is found and corrected, linear time is restored (right).