MODIS: No News, More Pictures

Still no change in our heater/cooler/spaceview door problem. Basically,
we’re waiting for permission to nudge the spaceview door open a little
bit.

Meanwhile, I’ve found some more places NASA is hiding pictures from MODIS
on the web. Why they don’t feel the need to tell us about these, I do not
know. At any rate, I’ve found two good sites. The first is

http://modland.nascom.nasa.gov/gallery

This is the web site for the MODIS Land Science Data Team. It has the best
all-MODIS, all-the-time, image sets to date. There are also related MODIS
land, sea and air web sites at the following locations, but the one above
is the best:

http://modis-land.gsfc.nasa.gov/
http://modis-ocean.gsfc.nasa.gov/
http://modis-atmos.gsfc.nasa.gov/

If you go hunting around those links, you’ll see all the different things
they use MODIS data for. We sure are keeping a lot of scientists and grad
students busy.

The second cool NASA site is the image gallery at the Earth Observatory
site:

http://earthobservatory.nasa.gov/Newsroom/NewImages/images_index.php3

This site has lots of cool images from a bunch of different instrument,
including MODIS, TRMM, Landsat 7, and SeaWiFS, all of which were built
here in Santa Barbara, and plenty others as well. This page has links to
twelve other pages, and each one has 8-10 pictures on it, so that’s a lot
f pictures. When you click on a picture, you get a larger version of the
picture and “a paragraph on the back each one explaining what each one was
. . .” (Who knows what that quote is from?)

While hunting around this site, I found a good writeup about some of the
problems we had getting Terra launched and operating. It’s a look back by
Kevin Grady, the Terra Project Manager, so this is as close to the horse’s
mouth as you can get. If you’re feeling cruel, you can go back to my old
updates where I described the problems and see how often I got the story
wrong. Check it out at:

http://earthobservatory.nasa.gov/Study/LearningToFly/
Another fun reason to hit that article is that on the third page, there is
an artist drawing of Terra from the space side, so you can see the door we
think is misbehaving. In the drawing, in the lower right, near the green
arrow, you’ll see our spaceview door in its open position. It starts out
flush against the spacecraft and opens downward. There are two dark
circles in the doorway. The one on the left is the hole that our telescope
look out to see deep space for calibration. The one on the right is the
“salad bowl” cooler I described last time.

I imagine NASA might update this site fairly regularly, so don’t wait too
long to check it out.

MODIS Heats up.

Well, we’ve encountered our first real and for true MODIS problem. About two weeks ago, we noticed that the heater on our cooler turned off.

No, that’s not a typo, we actually have a heater on our cooler. Let me explain.

Some of our detectors work fine at room temperature, but others have to be kept very cold in order to function properly. Cold in this context means about 83 degrees Kelvin, which is equal to =190 degree C, or about -310 degrees Fahrenheit. There’s two ways to get that cold in space. One is to use a mechanical cooler, which is kind of like what’s inside your refrigerator or air-conditioner. These do a very good job but have some drawbacks. The first is that they take power to operate (those of you currently paying your summer AC bills get the idea). The second is that the mechanical pump causes lots of vibrations (think about the rattling old fridges make when they start up or shut off). These vibrations have to be islolated from the telescope so as not to affect the view of the earth.

The other method for cooling is to use what’s called a passive cooler. This is basically a big, shiny, salad bowl pointed out into open space. You let the natural cold of space keep you cold. This has some problems of its own. For one, you have to make sure the thing is always pointed at space. It’s basically very good at transferring heat, but if it’s not pointing at space, then instead of draining a small amount of heat from our detectors into the big cold sink of space, it would deliver a lot of heat from the sun, or even reflected off the earth, INTO our detectors. Do this long enough and you can permanently damage the detectors. Another disadvantage of passive cooling is that space is even colder than we want so in order to get to just the right temperature you design the salad bowl to be just the right size abnd shape to get you a little bit colder than you want, then add an electronically controlled heater to bring it up to just the right temperature.

So, that’s why we have a heater on our cooler. Simple, right?

Q1: So what’s the problem?

A1: The heater on our cooler turned off, and we started warming up.

Q2: Wait a minute, if the heater turned off, shouldn’t you start cooling off?

A2: Only if the problem is in the heater. We think the problem is in the cooler

Q3: Huh?

A3: Think of it this way: if the heater in your home breaks down during winter, you get cold, but that’s not what’s happening here. Think if you left your thermostat set to 70 degrees all winter. Your heater would run pretty much all day and night. As spring comes and turns into summer, your heater will be on less and less. First, it will turn off during the hottest part of the day, and back on at night. As the days get warmer it may only be on for a short time in the coldest part of the night. Eventually, when the temperature never gets below 70, your heater shuts off for the rest of the summer. That’s essentially what’s happening now.

Q4: So, why is that a problem?

A4: You don’t have summer and winter in space. It’s supposed to be cold all the time. But our cooler is starting to warm up, even though our heater is shut off.

Q5: Any ideas why?

A5: Several. First we thought it might be contamination of the cooler, caused by exhaust from the thrusters, as the problem seemed to start after one of our thruster burns which put us into the correct orbit. That didn’t seem to square with the observed temperature, as it seems to have an orbitial variation.

Q6: I suppose this is where I’m supposed to ask what an orbital variation is?

A6: Yes, thank you. An orbital variation means that as the satellite orbits the earth, the temperature of the cooler seems to rise and fall in different parts of the orbit. If there were any contamination on the cooler, the temperature would go up and stay up.

Q7: By the way, if the problem started back in February when the orbital maneuvers were taking place, how come they’re just noticing it now?

A7: Well, while the cooler was warming up, our heater had to work less and less hard to keep the detectors at the right temperature. Since the thing we really care about is the detector temperature, we didn’t notice the problem until the heater shut off for good and the temperature kept rising. Whoops!

Q8: So if it’s not contamination, what is it?

A9: We think that the space view door isn’t all the way open. The space view door covers the cooler during launch. If for some reason it had started to close, we could get some reflections off the door back into the cooler.

Q10: How do you tell if the door is open or not?

A11: There are little switches on the door that close an electrical connection when the door is open. We saw the switch connect when we opened the door, and it amount of time it took to open the door was what we expected. So we thought we were OK.

Q12: Does the switch still say the door is open?

A12: Yes, but the switch will read OPEN when the angle is anywhere from 75 to 95 degrees away from full the closing position. So it could start to close just a little and we might not know it.

Q13: How could the door close without you commanding it to close?

A13: Good question. We have people hypothesizing all sorts of strange things I won’t bore you with. Let’s just say that nobody has come up with a smoking gun.

Q14: I remember from an earlier email that you said the software to control the doors was one of the longest, hardest, most intricate part of the software and that you were really glad it seemed to be working.

A14: Quite true.

Q15: So you must be really annoyed that there seems to be a door problem, aren’t?

A15: Yes . . .

Q16: And I bet you’be been spending time in lots of meetings with people who don’t understand how the doors behave who want to second-guess your designs, and ask irrelevant questions?

A16: What’s your point?

Q17: So you must have that same irritating feeling you get when you get bubble gum stuck to you shoes and even once you get almost all of it scraped off, you can still feel the little sticky tug every time you take a step?

A17: Stop that! Aaarrgghhh! Leave me alone!

Q18: So what do you do?

A18: Well, about 15 minutes after hearing about the problem, I asked the mechanical engineer “If the door is close to where we think it should be, how many more steps open would you feel could be safely taken to try and get the reflections off the cooler?” He answered 100 more steps should be safe. That was 10 days ago. Since then we have presented all the data to our management, and to NASA, who don’t want to do anything in a hurry until they feel they truly understand the problem and are sure our solution won’t create any new problems. Unfortunately, with this problem, there’s not much more data we can get. In my opinion, we just have to try and see what happens.

So hopefully, sometime in the near future, we’ll try to nudge the door a little more open, and see if our cooler starts getting cold, so our heater will start heating

Got all that?

JAMI launch

This file contains various emails I sent around to family just before and after the  JAMI launch:

Date: 2/34/05

Subj: Launch from Japan, T-2 days:

Well, I haven’t bored you all with one of these in a while, so yes, we’re approaching another launch, this time from the Japanese Launch Center in Tanegashima, Japan.

The satellite being launched is the Multi-functional Transport Satellite-1 Replacement (MTSAT-1R). There are 2 basic missions for this satellite. One is the Aeronautical mission, which is a radar which will be a big part of the Japanese Air Traffic Control System. I don’t really know much about this part. The other mission is the Meteorlogical mission, which is the part I worked on. It is basically a weather imager with some extra sensors for science data as well.

The original MTSAT-1 was lost when the H-2 rocket built by the Japanese blew up on the launch pad in in 1999. We didn’t build the imager on that one. We bid on the imager for the replacement for that satellite and won that contract, which is the thing we’re sending up now. We managed to go from a blank sheet of paper to a delivered instrument in 39 months, which is a remarkably short period for something like this. To give you an idea, the MODIS instrument that I worked on previously (which was more complicated
in a lot of ways, but simpler in some) took over 7 years to go from contract award to delivery.

The launch has been delayed for a while while the Japanese review why the H-2A rocket (which was the new improved version of the H-2 rocket), which also blew up on it’s maiden attempt. So they have been checking it out very carefully. The launch had been scheduled for early in the morning CA time on Thursday Feb 24, but bad weather in Japan has pushed it back to early morning CA time on Feb 26th.

I will not be attending the launch. I will be watching a live Web broadcast, which you can too, from the web site

http://mtsat1r.rocketsystem.co.jp/index_e.html

It’s a pretty good web site, and even had a web cam that shows a live picture of the launch pad. Unfortunately, since the Japanese are 17 hours ahead of California, and 14 ahead of the East Coast, if you look at it in the middle of the day here, all you see is a black box, since they don’t have lights on the launch pad at night. D’oh!

Anyway, if you browse around the web site, you’ll see our instrument referred to as JAMI or as Imager, depending on which part of the site you hit.

As with the other instruments I’ve worked on, they really won’t be doing much with our instrument for the first 7-10 days, but they will be turning it on and doing domw basic checkout prior to cooling our detectors and starting to take pictures. So assuming all goes well with the launch, I’ll be sending along some interesting stories about this mission, and about just how many corners you have to cut to build a brand new instrument and deliver it in 39 months.

Meanwhile, while I’m supporting the early on-orbit JAMI checkout, I am working on another program for NASA, which is rather behind. So, I have the joy of working double shifts for awhile. This actually works out OK, since most of the work in Japan is done during regular business hours in Japan, which is about 5PM to 2 AM in California. So I’ll work a little on each project on each of first and second shifts, then go home late.

It also explains why I’m up writing emails at 2AMin the morning.

Joe

 

Date: 2/26/05

Subj: JAMI has left the Planet!

 

Joe’s Late Night Launch Log:

Well, at about T-40 minutes, the communications system between the launch vehichle (the H2A rocket) and the ground station reported an error. Of course, they didn’t put this up on the web site or on the video feed until after the launch time had come and gone, so a few of us were left sitting here in the middle fo the night wondering what was up . . .

. . . and at 12:40AM CA time, the web site reported that a new launch time of 1:25 AM has been selected, which is just about at the end of the launch window, as I recall . . . this is 6:25 PM in Japan so we can watch the launch pad get darker on the web feed as we wait.

Fortunately, I have some boring work to keep me busy . . . ZZzzzz…… I also have time to run out and get some coffee (Hmmm… coffee….)

1:00, and all is well . . . .

1:10, and they confirm the new launch time of 1:25 . . . safety checks all confirmed, and the weather has been reviewed and confirmed acceptable . . .

(I’d like to interrupt this launch log to point out that the original MODIS, on the Terra spacecraft, passed its 5-year operational mark this week. That was the original design lifetime for the instrument, so we can now claim it as a complete success. There were a number of interesting problems over the five years, and it helps with future sales if we can show that we meet our design lifetimes. With luck, we can get several more years of operation out of it. There were unexplained (for a while) computer dropouts, coolers that didn’t cool, heaters that didn’t heat, and even a mistake by NASA operators that resulted in a door mechanism breaking (remind to tell you about that one sometime). Through it all, we have been generating science data continuously for 5 years with only about 1 month of total image outages, for a coverage rate of over 98%. Please excuse me while I pat myself on the back. We now resume our origianlly scheduled countdown in progress.)

1:20, and the countdown is progressing . . . the sun has set in Japan, and they’ve turned on floodlights to illuminate the spacecraft before the really really big candles get lit up . . .

T-120 seconds, and the launch vehicle switches to internal power (it’s now operating on its own with no support from the ground)
T- 60 seconds and counting . . .
T-30 . . .
T-10…9…8…7…6…5…4…3…2..1..LIFTOFF!

It cleared the tower as planned and is heading southeast over the Pacific, being picked up by the tracking stations along the way . . .

Five minutes past launch and everything looks good . . . fairing jettisoned successfully … on the web link they are showing video from a camera on the outside of the rocket as it rises up over the earth, and as the rocket gets higher, the sun comes out into view from from behind the earth . . . VERY beautiful! I will definitely look for copies of the video on the web and let you know. Solid rocket boosters separating normally, first stage running normally. . . . first stage engine cuts off normally, and sepates . . . second stage starts on schedule . . . trajectory looks good . . . . 270 kilometers high, velocity 5.1 kilometers/second, and accelerating . . .

This satellite is heading for geosynchronous orbit, which is about 22,000 miles high. It is too wasteful of fuel to try and go straight up to that high an orbit, so instead, the launch rocket will leave the satellite in
what is called geosynchronous transfer orbit, which is much lower, Then the spacecraft slowly raises its height a little bit on each orbit. As the orbits get higher, they take longer. By the time they get to the proper
height, the orbits will take exactly 24 hours. So from the ground, the spacecraft will look like it is fixed above the equator, south of Japan. This whole process will take about 9 days. During that time, we will turn on our instrument (JAMI) and start the process of outgassing, which is where we turn on heaters to burn off any remaining traces of atmospheric gasses and contaminants. That process also takes about 10 days. At the end of that, we will pop open our door and start trying to get imagery data to the ground.

T+15 miuntes, the rocket goes into a coasting phase.
T+20 minutes, and the tracking station at Christmas Island should be picking the spacecraft up soon . . . and it is . . .

When we first open up, we don’t start out with taking pictures of the earth right away. We start by trying to locate stars. We point our mirrors to get a star of known brightnes to drift by our field of view at a particular time, and the exact time and position of the star crossing helps the spacecraft team determine the exact position and orientation of the spacecraft. This is needed in order to control the pointing of the spacecraft. This is much more critical for this mission compared to the earlier ones I’ve talked about, because from such a high altitude, even slight errors in pointing will throw off the images on the ground. Our instrument can take a picture of the whole earth in about 22 minutes, sweeping the telescope back and forth, east-to-west then west-to-east, and gradually moving the swaths south over the earth. We can also take pictures over much smaller regions at a faster rate, but we can’t do that until we get the pointing just right. So, after we open our door, we have several weeks of testing and alignment planned.

At 2:06, the H2A separated from the spacecraft, and they cut to a video of the people in the launch center going absolutely bonkers. Cheering, backslapping and handshakes all around. After the launch failures the
Japanese have had with their rockets, including a failure of the first H2A (this one was only the second one) this is big win for them.

Well, it’s closing in on 2:15, and I think that’s enough for tonight. I still have to send this to my email account at home, and then forward it out to all you so I still have a ways to go. I’ll keep you all posted with any new and exciting developments as they occur, but for now, everything looks really good.

Joe
Date 3/4//05

Subj: Good News, Embarassing News

 

Well, JAMI has been turned on and is basically working fine (good news), but we’ve already uncovered the first error in the software I wrote (embarrassing news).

The Good Stuff:

After a perfect launch, the orbit raising (when the kick the satellite up to geosynchronous orbit over the course of about a week) went smoothly. The Japanese website that tracked the launch has put up some cool pictures of the launch which you can see here . . .

http://mtsat1r.rocketsystem.co.jp/live/internet_e.html

There are three rows of thumbnail pictures along the bottom of four pictures each. The second row shows the view of the sun coming into view over the edge of the earth. Now all we need is the video . . .

Also note the lower row which shows the spacecraft separating from the launch vehicle. The second one from the left is a good view of the spacecraft as it pulls away. The white thing sticking out of the bottom is a piece of our instrument called the sunshade. Unlike the MODIS instrument which was in low earth orbit, and so never had to worry much about looking straight into the sun, JAMI will be much higher (22,000 miles for JAMI, vs. 435 for MODIS). At that height, when the sun passes behind the earth each day around midnight Japan time, we run the risk of having the sun shine down our telescope and start heating up our detectors. You don’t even have to shine straight down the telescope to cause problems, even stray sunlight can be a problem. So we added a big box around the front end of our telescope to cut down on that.

We turned on JAMI for the first time today, and began outgassing. This is the process where we heat up the instrument from the inside out, boiling out any remaining gasses, dust and water vapor from the atmosphere. We do this for about a week before we try taking any images. The operations team reported that everything looked fine, and that the heaters were starting to warm everything up just the way they should, and all the telemetry looked good, except for one little thing . . . . .

The Embarrassing Stuff

They also reported that the instrument telemetry was reporting out that it was in imaging mode which is what it reports when you start taking pictures. It should have been reporting that it was in outgassing mode, which is in fact what it was doing.

Q1: So what happened?
A1: Short answer, the bit of software I wrote which reported out this status had a really basic error in it, the kind of thing I got sick of reminding my students to watch for when I was teaching programming a few years back. Basically, no matter what the state of the instrument, the JAMI will report
out that it is in IMAGING mode.

Q2: So how big a deal is this?
A2: It’s more embarrassing than anything else. I haven’t yet figured out if it will be easy to patch in a fix for it, and I’m not sure they’ll even bother, since patching software on orbit is a big deal, and has to get lots
of people to approve it, and so may not be worth the hassle of fixing.

Q3: How come nobody noticed this before? They found it pretty quick on orbit.
A3: Well, the usual answer in these cases is that it wasn’t properly tested. This particular piece of telemetry was added relatively late in the game, and during ground testing we had other more informative ways of verifying the system state, so we never looked at it that hard. Besides, it was such a small simple change that I just KNEW it was going to work . . .

Q4: So, is the inadequate testing one of the corners you mentioned had to be cut to get the instrument delivered in 39 months?
A4: Next question

Q5: But that’s not really an excuse is it?
A5: Really, I mean it, NEXT QUESTION!

Q6: What’s next for JAMI?
A6: We outgas for 8-9 days, during which nothing much happens. The ops team will try loading up the daily schedule of imaging tasks, because they have some concerns about loading large tables, which these are.

Q7: Since you’re in geosynchronous orbit, why do have a different schedule every day? Don’t you just a picture every hour on the hour or something like that?
A7: For pictures of the earth, thats exactly right. However, interspersed with all the earth images, we scan off to the side every so often to take a look at a bright star. Because the earth wobbles a little in it’s rotation,
the time and location to look for a particular star changes every day. Also, because the north pole isn’t exactly perpendicular to the plane of the earth’s orbit, the sun and moon drift through our field of view at a different time and location every day. All that information has to be uploaded to us each day.

When we are just about done with outgassing, we pop open our door (basically the lens cap for our telescope) and start trying to take visible images. We won’t have the infrared (IR) detectors cooled down yet, but we can get started checking out our ground system. which is a story for a future email . . .

Anyway, its up and running, and we’ve proven that we can do some of the things we have to do to investigate unexpected events, so that’s all to the good. It’s just that Computer Programming 101 error is something that I’m sure I’m going to hearing about for a good long while.

Oh well. at least I’m off overtime for a while. . .

Joe
Date: Saturday, March 12, 2005 11:32 PM
Subj: Images coming soon . . .

Well, we’ve just about finished our outgassing for decontamination, and tomorrow we open the door coverin our telescope. This happens about 1:30 in the morning Japan time, which is when the sun is behind the earth (from Japan’s point of view) so it’s when the sun is most directly shining at our door, but with the sun out of the field of view of our telescope. We do this so that our door mechanism is as warm as possible when we open. The mecahism is a parafin heater, which melts the parafin in the latch, which allows the door to spring open. It sounds kind of Rube Goldberg-esque, but its actually the most reliable way of doing this sort of thing. Motors and electonic switches and that sort of thing (which always look cool on Star Trek and in James Bond movies) are actually more subject to failure.
Of course, with the sun shining on our door, and Japan in the dark, we can’t take any pictures with our visible light detectors. For that we have to wait until daytime over Japan, which is evening in California. Our infrared detectors take useful pictures at night, but they don’t work until we turn off the heaters (which are still outgassing for another day or so) and turn on our coolers. These will cool the detectors down to about -190 degrees Celsius. That won’t be for a few days. So, in the meantime, we work with just the visible bands. As I mentioned earlier, the first thing we do is try to locate some stars to help with the spacecraft pointing, but they may try to get an earth image out early to check out the ground system.

Oh, did I forget to mention the ground system?

You see, our system is replacing the original JAMI, which was lost due to a launch failure. The Japanese had a bunch of equipment ready to process the data which came out of the original JAMI. In order to deliver the system as fast as we did, we had to re-integrate existing subsystem design components from some of our other instruments, with a minimum of changes. In this context, “re-integrate existing subsystem designs” means pretty much the same thing as “Slap that puppy together fast. If it doesn’t fit, hit it with a hammer. If that doesn’t work, hit it with a bigger hammer. If it breaks, get the duct tape.”

One of the things we couldn’t do was get our instrument to generate data in exactly the same format as the earlier instrument. This was mainly due to the fact that our instrument actually has better resolution than the old instrument. And as much as the Japanese would have liked to have the higher resolution images, the ground equipment they spent a lot of money on wasn’t set up to process it.

So, we had to provide the Japanese with a piece of equipment which reads in our data, slices it, dices it and spits out something that looks like the low resolution data the original instrument was going to generate.

This is an easy thing to say, but difficult to do in practice. There a lot of reasons why, but the bottom line is that as late as a week before launch, we still didn’t have it working as well as we needed. This wasn’t really
my area, but since most of the engineers who worked on JAMI have moved on to other projects, I got stuck helping out trying to figure out what was wrong. Our customer was getting worried, since if we couldn’t get the images straightened out, they wouldn’t be able to do the pointing alignment which is the first step in getting the whole thing operational. So rather than trying to find root causes, they told us to force the system to rotate the data to remove the errors that we were seeing (“Hit it with a bigger hammer”). So, while we we’ve been outgassing, I’ve been working with whtat’s left of our team going over every equation, every number, and every line of code in the software to try and find what’s wrong, and we made a lot of progress this week. So I feel much more confident about how the images will look today than I did at launch time. And yes, it is hard to type with all my fingers crossed.

Anyway, the upshot of all this is that they may try to process a visible light image as early as tomorrow night. I’ll let you know how it goes.

Steve Nolan has pointed out to me that the Japanese web site which has the launch pictures also has a video of some of the best parts of the launch and early flight. It’s about a 15 MB video file, so if you have a slow modem connection, it is probably too large to download. If you have a high-speed connection, or don’t mind an hour-long download, you can get it here:

http://www.rocketsystem.co.jp/gallery/movie_mtsat1r_e.html

Joe

Date: 3/15/05

Subj: JAMI Door Open, First Picture Taken, and Guess What?

 

The earth is round! And is covered with clouds!

The door opening went according to plan, and they have started taking images. So far, they’ve sent along one picture that covers about one third of the earth, taken mid-afternoon Japan time, so about 3/4 of the earth is visible. This was processed through our ground system, and shows some of the problems I mentioned earlier. They also sent along the raw data for a full earth image. That image was taken just before sunset so there is just a sliver of visible image along the western edge of the globe. Our ground system only got a third of the way through that before shutting down. So we’re trying to figure out why that happened. I’m busy grinding through humongous data files (550 MB for a full earth image) trying to see if any of the data coming out of the instrument was bad. The good news is that so far, it looks like the instrument is working fine and the problem is either in our part of the ground system or in the Japanese equipment. We have a few more days to shake out the bugs, then over the weekend or early next week, we cool down the detectors to start taking infrared images.

Our end customer for this project is the Japanese Mapping Agency (JMA) and they have asked that we keep the images in house until they release the first official images, so I won’t be sending them around just yet. But, if they don’t do something official by the end of the week, I might (accidentally) reduce one of the images to some thing that can be reasonably email-ed (for no particular reason), and then (carelessly) send it from work to my home email account (whoops!) and then (inadvertently) drop it in some email list or other (completely by mistake).

Joe
Date: 3/20/05

Subj: What IS this?

 

Why look, its an email from work. Funny, I have no legally provable memory of sending this to myself. I wonder what it is. Appears to be a picture of some kind . . .

Oh well, I guess I’ll just delete it . . . DARN, now I seem to have clicked the “Forward to” button instead of Delete” . . . and by coincidence the name at the top of the list is my space email group . . . I’ll just hit Cancel, that way I won’t pester anybody . . .

DARN IT AGAIN! I hit “OK” instead of “Cancel”. I’m just all thumbs today. Which is not surprising as we’ve had a lot of late nights this week. We’re still on schedule to go cold early next Tuesday, Japan time which is Monday afternoon here.

Still, I’m better off than the guys supporting the spacecraft in Japan. I get to go home and sleep in my own bed at night. They get to go to a hotel in a foreign country, and work with the Japanese customer looking over their shoulder all the time. Speaking of whom, they still haven’t released any pictures yet, they are probably waiting to get some IR pictures to go with the visible light images I am in no way sending to anybody as far as anybody’s lawyers know.

Well, deleting this whole email seems wasteful now that I’ve already started it. I’ll just be sure to delete the attacment (whatever it is) before I click “Send”.

If I don’t get some sleep soon, I’ll probably be so groggy that I’ll send along some closer in pictures taken from the same image, that, as I said before, I have absolutely no knowledge of.

Since I have no knowledge of these alleged pictures, I can’t be sure that they have closeups of places like India, Sri Lanka, and Australia, or even a bit of Japan, at least those parts that aren’t covered with clouds. Um, if there are any clouds. Not that I’d know what clouds look like from any satellites that might be up there.

Now, what was I going to do before I clicked the “Send” button? I’m sure there was something. Oh well, it probably wasn’t all that important.
Joe

Aqua Launch Delayed

Here We Go Again, Part 1

Due to various problems, the launch of Aqua has been delayed by one week, to April 25th. Here’s some of the things they’ve been looking at:

1: There is a concern about one of the batteries in the spacecraft. As the spacecraft orbits the earth, it keeps its solar panels facing the sun, which provides power for everything on the spacecraft. However, when orbiting over the night side, the spacecraft operates off of batteries which are charged up when orbiting over the day side. They are not sure, but they are concerned that one of the batteries might be discharing too quickly. There are spares, but they don’t like to launch already using up spares.

2: There is a box containing some kind of electronics on the outside of the spacecraft, near the viewport we use to look at the sun for calibrating our science data. They repositioned this box for some reason, and we think it is now in the way of our viewport. We have given them a clearly defined “keep-out” zone, and they claim they are outside our field of view, but our guys don’t think so. So, we are arguing about that.

3: (and this is the fun one) They recently modified something on the Delta Rocket which is our launch vehicle. The first couple of launches in the modified configuration measure significantly larger shock as a result of Main Engine Cut-Off than was expected. About FOUR TIMES larger. So NASA asked everybody to review the their test history, to see if they felt comfortable with shock loads four times higher than specified. The machanical engineer on our program redid all his analysis and it looks like we think we will probably be OK, but with no margin for error. The most critical part of our instrument in this regard is the structure on which we mount all of the optics leading out of the back end of the telescope. There are very carefully positioned mirrors, lenses and filters, which feed into the detectors which actually create the image. So we are pretty concerned about that.

Anyway, we actually started fueling the spacecraft this morning. Anything involving rocket fuel is a dangerous operation, and they don’t start on something like that until all the questions like the ones above had been answered to everyone’s satisfaction. So the new date looks pretty good.

Here We Go Again, Part 2: Terra Goes into Safe Mode

Back when we first launched Terra, you may remember that we had problems getting the spacecraft into just the right orbit. One of the problems we had was that Somebody left on an alarm when they fired the maneuvering thrusters. Those alarms are supposed to go off if the spacecraft starts rotating unexpectedly. Since rotating the spacecraft was what we wanted to do, they should have turned off the alarm. The alarm shuts down all the instruments and gets them to close their doors.

Well, over the course of the last few years, the spacecraft’s orbit has drifted slightly, as happens with most spacecraft. The Flight Operations team has been monitoring the drift, and decided that they wanted to adjust it slightly to get us back on track. They remembered to turn off the alarms this time. The performed the maneuver properly. They turned the alarms back on, and they immediately went off, sending all the instruments into safe mode, and we closed our doors. It seems that after the maneuver was complete, the spacecraft continues to wobble back and forth for a little while. So they should have waited a little longer before turning the alarms on. Because NASA takes unexpected events nice and slow to make sure they fully understand what happens, we left our door closed for about a day. We opened our cooler door first, to let our cooler get back down to the operating temperature before turning all the electronics on. About three days after we got shut down, we got everything turned back on, which leads directly into:

Here We Go Again, Part 3: MODIS Computer Won’t Start Up

One of the two on-board computers in the MODIS instrument has a known problem where it gets stuck rebooting every few seconds. We have dealt with this in the past by placing the instrument in a reprogramming mode, and patching the program to ignore the signal that it telling the computer to reboot. At least that’s how we dealt with it the last time they turned us off by mistake. It turns out that the problem has gotten worse over time, and the same problem is now occuring in the reprogramming mode. We did find that when we first turned on, we could keep the computer up and running for about a minute, then it started rebooting. As the temperature of the system rose, we found another window of about a minute where the problem goes away. We used these brief gaps to first patch the reprogramming mode to ignore the error. This kept us up and running long enough to get the other patches in. Then we were finally able to get the normal code up and running. But we are concerned that degradation in performance may continue so that we won’t get any windows long enough to reprogram the computer. So we have reccommended to NASA that the changes be installed permanently into the board. This requires powering the memory chips in a certain way. This mode of operations makes the memory chips more susceptible to radiation damage, which would make the chips unreadable. As a result, we could lose the whole board. So we are researching just how likely we are to damage the chips in this process, because we have no reason to believe that we will be able to pull off our reprogramming tricks if the board continues to degrade. On the up side, we did finally figure out the underlying cause of this problem and was able to fix it on the MODIS going up on Aqua. If I can talk NASA into risking the reprogramming, I might not have to drop everything I’m doing to help them get the thing up and running every six months. That will be very nice. Plus, it will be nice to have a second instrument up there that doesn’t have this problem at all.

So it lookslike we are good for launch on the 25th. I’ll keep you posted.

Joe

Aqua Launch Date Set

Date: Tue, 26 Feb 2002

The subject says it all.

The second MODIS instrument, now fully reassembled and reinstalled on NASA’s Aqua spacecraft, is scheduled for launch on April 18th.

Now, everything that launches out of Vandenburg Air Force Base always flies south. This is because when you fly south from Vandenburg, the first land you hit is Antarctica, so that’s the safest direction to launch. To get the Aqua spacecraft into its proper orbit, and still launch to the south, they have to launch between 2:55 and 3:05 in the morning. Since Vandenburg is northwest of Santa Barbara, that means the rocket flies right past us on the way to outer space (well, on the way to low earth orbit, anyway). So rather than pile into a bus up to Vandenburg, like I did for the Terra launch (which launched at the much more civilized hour of 10:45 AM) I’ll just step outside at the right time and watch the show. If you’ve never seen a nighttime launch, you’re really missing something. They launch missile defense tests out of Vandenburg, and one last year was at about 9:00 at night. That particular rocket left a bright green trail and lit up most of the sky. So unlike the satellite watching that I sometimes ramble on about, it won’t be hard at all to spot the rocket.

Did you notice up above that I said the second MODIS was now fully “reassembled”? During Aqua spacecraft testing last fall down in Los Angeles, we a serious failure of the electronics in MODIS.  If you recall from one of my earlier emails, one of the on-board computers on MODIS would continually restart when it was cold. We could never figure out why, but we went ahead anyway because the temperature range at which it occurred was well below the temperature at which we expect to operate.

Well, it started happening right in the middle of the operating range. So we had two choices, the same two choices we had before: tear it apart and hope we can find the problem, or ignore it and hope it goes away. This time we decided to tear it apart.

Because this moved way up the list of concerns for the spacecraft, NASA sent out a team of three electronics guys. These were three really good engineers, but since they were unfamiliar with our electronics, somebody had to sit with them, explain how our designs are supposed to work, what the problem is, what we’ve already done to try and figure it out, and basically answer any questions they might have. Ideally, this would be the engineer who designed our computer board, but he’s long gone. The guy who replaced him is good for answering specific questions about the electronics design, but as far as the behavior of the computer and the software goes, I’m the best guy still here.

So the team was out here for about 3 weeks, and we finally traced it down to a particular part of the design. Here’s the short version (the long version was a 50-page report): Any tme you link together a bunch of electronic components and send signals between them, you can get a electronic reflection in the wire connecting the components. The reflected signal interferes with the original one, and you get garbage. Getting rid of these reflections (a process called termination) is a standard design problem. Anyway, a change we made to the computer board design way back in 1994 altered the termination of part of circuit board, and corrupted one signal just enough to cause problems, but only when we go cold. When you change temperatures, the internals of electronic component speed up or slow down just a bit, and you have to make your designs work with this uncertainty. Ours did, but not with the corrupted signals caused by the bad termination.

Got all that? And remember, that’s the short version.

The good news was that we only had to cut one signal line and replace two components to correct the problem.

The bad news was that we had to do this on four different computer boards.

The worse news is that when you make any changes to equipment that is supposed to fly in space, you have to repeat as much testing as you possilbly can. For us, that meant performing the acceptance tests at hot and cold temperatures, which was about a week for each of the boards. Again, we hadn’t run these level of tests in several years, and I was the only person still around from the last time we ran them, so even though electronics testing isn’t really my area, I got stuck helping out with the testing.

We got it all done by mid-December, which was important so that we could get it all put back together in time to support the final testing of the spacecraft before they shipped it off to Vandenburg.

Meanwhile, the Terra spacecraft is purring along nicely. The same termination problem I described above is present on the boards up there, and we can’t fix them. So far, only one of the boards is exhibiting the problem, and for that particular case, we have a software patch that is working so far to prevent the error from crashing the computer when it occurs. The problem here is that it is occurring more and more frequently, about 100,000 times per day. and we are starting to wonder at what point the time we spend clearing the error starts to impact the perfomance of the instrument. But for now, it seems to be handling it without problems.

We got a bunch of good press recently on the evening news about Terra/MODIS collecting the most detailed measurements ever made of sea surface temperature all over the globe. You can read all about it at:

http://earthobservatory.nasa.gov/Newsroom/SST/

I’ll keep you posted on the launch date, but it looks pretty firm for now.

Joe

 Terra/MODIS Update

. . . Or, How to Have a Really Busy Two Weeks in Aerospace

When we last left MODIS, the heater on the cooler wasn’t heating, mainly because the cooler on the cooler wasn’t cooling. Got that? The two theories were that there was some kind of contamination on the cooler, or that the Spaceview door had somehow managed to close itself a little bit, allowing reflections of Earth and Sun back into our cooler.

After much hand-wringing they decided that the door possibility was most likely, mostly because it was something they could try to fix without interrupting the flow of images to the ground. To try and clear the contamination, we have to go back into outgas mode, which means we turn our detector off, as apparently it’s bad to have then turned on when the outgas heaters kick in to try and boil off whatever crud might have collected on the cooler.

So after asking me endless question about how this could have happened (which involved longer and longer messages which basically amounted to “Beats me”, they decided to nudge the door open a little bit more.

No effect whatsoever.

So now they want to outgas the cooler, but can’t figure out when to interrupt things for the 3-4 days it will take to heat up and then cool off. But they’ll get back to us.

All of which is fine with me, as I have been spending most of my time on two other things anyway. The first is the second MODIS now being put on the Aqua spacecraft down in Los Angeles. The second is preparation for A Big Review of the software for JAMI, new weather satellite I’m working on. Remember that one? Last time I mentioned that we were hoping to win a second one, and that we had an inside track for that since we had just beat our competition for the first one, and we figured that they’d want two that were the same, rather than two different ones. Well, we didn’t win the second one, and the only thing that we can figure out is that our competition lowered their price a bunch and also teamed with a Japanese company, where ours was a pure US effort.

Well, at least we got the first one, which was an unexpected bit of extra business anyway. At any rate, I’ve been letting the work on JAMI get really far behind as they’ve been have trouble with the spacecraft down in Los Angeles, and they haven’t been able to collect the data that we are sending to them. So I’ve been spending most of my time analyzing huge data files and sending them reports on what data is missing where.

As a result, JAMI has been falling further and further behind. But I had cleared my desk for the week before The Big Review, and planned to dedicate my full effort to accomplishing something for the review. I was making good progress as of Friday Aug. 4th, and I decided to enjoy the weekend and then finish up strong for The Big Review.

Saturday (Aug. 5th) was lots of fun. One of my favorite local establishments was having its 5 year anniversary, and so was holding a classic car show in its parking lot at 5:00, a pot luck at 6:00, and a band at 9:00. The classic car show was great, with about a dozen huge cars from the 50’s and 60’s. The food was great. My contribution was a crockpot of vegetarian chili, which I made without repeating the Hideous Garlic Tragedy of 2000

(but that’s another story). The band was a surf band called the Neptunes and they played three long sets well into the wee hours. After they were done, the band joined the party. For about 20 minutes the two guitar players were sitting out on the patio working out some kinks in a new song they’re working on. It was fascinating watching the back and forth between the two of them as they tried different guitar lines for the song.

Anyway, the party went late. I got home at about 3:00 in the morning, looking forward to a nice long sleep.

When I got home, my pager was buzzing, and there were 3 messages on my answering machine. (For those of you keeping track, the Really Busy Two Weeks in Aerospace begins about here.)

It seems that people operating the MODIS instrument on Terra had been trying to contact me since about 7:00 because MODIS had begun acting strangely. It was hard to tell, but they said first that the image data was dropping out periodically and then after an hour of that, they thought our scan mirror had stopped spinning. I don’t know if I’ve mentioned it before, but the way our instrument works is that our telescope is fixed, and doesn’t move. In front of the telescope is a two-sided mirror which spins around, reflecting a wide scene down into our telescope. If the mirror doesn’t spin, we don’t get any pictures. So when they thought it stopped, they were worried that something was broken, so they shut everything off. In addition to not being able to get ahold of me, most of the rest of our team was out that night as that weekend was Fiesta here in Santa Barbara, which is a local Hispanic festival. As a result, they could only find 1 or 2 people, who figured better safe than sorry, and had them shut it down. The last message on my answering machine was from our program manager letting me know that they were asking everybody to come in for an 8:00 meeting the next morning to help figure out what was happening.

I dragged myself in to the meeting, and spent three hours discussing things with just our group. Then we got to talk to the people from NASA for another three hours discussing pretty much the same thing. Mainly, we asked them to send copies of the telemetry collected during the odd behavior. The telemetry is just a bunch of numbers that one of the on-board computers sends to the ground every second.

They pulled out just the numbers we were looking for, and I was able to figure out the mirror was spinning just fine (at least until they turned everything off), but that the second computer wasn’t functioning properly.

MODIS has two computers running in normal operations. The first one talks to the spacecraft, the second one controls the picture data that is sent to the ground. When the second one stops working, you get rather strange and misleading telemetry. Since I’m the guy that wrote the software, and I’ve been working with MODIS for about 8 years now, was the only one who had seen that type of behavior before.

What is happening is that the second computer runs for about 20-60 seconds, then reboots. It goes through its boot up process, runs for another 20-60 seconds and reboots again. Because it’s never up for very many seconds, it can’t really generate data the way it is supposed to.

In other words, “MODIS HAS FALLEN AND IT CAN’T GET UP!”

I figured that if they just turned everything back on it might come up normally, or at least be up long enough for me to dump pieces of its memory, which might help me diagnose the problem.

Part of the difficulty is that we are not in continuous contact with the spacecraft. There are several communications satellites orbiting the earth, and as Terra moves through its orbit it makes contact with whichever one is closest. Lots of spacecraft use these satellites, so access time is carefully controlled. Because Terra is a high priority mission for NASA, we get a lot of time, but it’s still not continuous.

As a result, we get about 20-30 minutes of contact, separated by 20-30 minutes of time when we are out of contact. When we’re out of contact, we try to figure out what to do on the next contact. (And sometimes I write email updates to family and friends, like I’m doing now!)

The good news is that we have a backup computer, same as the one that is misbehaving, which we can turn on if nothing else works. However, the people at NASA want to understand what has gone wrong, so that we are sure the same thing won’t happen to the backup if we can prevent it.

Anyway, we were able to put the failed computer into reprogramming mode, at which point we could reliably talk to the computer and see that it was basically OK, but any time we tried to run the normal program, the computer would start rebooting.

Also, we discovered that NASA had a major field campaign planned to start about Aug. 15th. This field campaign (called SAFARI) was planning to make all kinds of coordinated measurements of Africa. There is MODIS data from space, ground data collected by groups of scientists in the field, and air data taken from aircraft. So there are scientists spreading out all over Africa, and MODIS data is needed to coordinate it all. As you can guess, the scientists want to switch over to the backup as soon as possible. Meanwhile, we are trying to figure out what little software tweaks we can put in to try and diagnose the problem.

We also discovered that several other spacecraft had problems Saturday night, so it may be the case that there was more than normal solar radiation at that time. All electronics for space are designed to withstand the radiation environment in space, but there is always the possibility of a large burst at one time. On the same night we went SPLAT, the Hubble Space Telescope also reported some problems (I’m not sure what, but they cleared up pretty quickly) and the QuickScat imager also shut down briefly. So NASA has their scientists trying to figure out if it was extra noisy out there that night. So far, no word.

Oh by the way, I had to interrupt all this to take a few hours to give my part of The Big Review on JAMI. It was hectic, and there was no hiding the fact that we are falling behind. I was supposed to be on a business trip to Virginia for JAMI this week, but I sent my boss in my place.

During all this, we found time to finally outgas the cooler and it seems to have worked! We are now back down to our normal operating temperature. We aren’t really sure what could have gotten on the cooler, but the best candidates are exhaust from all those small burns we did to get into our ideal orbit, or crud outgassed from other instruments over time (we outgassed for 5-6 days at the start, other instruments did much less).

Anyway, people are very happy about that.

So I spent most of last weekend coming up with clever little software patches to try and diagnose the problem. Of course, in the middle of this I had to run down to LA for the evening to help with the testing down there. It went sort of OK, but the folks at TRW (who are building the spacecraft) are really stretched thin, and didn’t have quite all the people they needed there on second shift. At least we got through the tests of our three doors, which is why I was there. So I came in Monday morning, (the 14th), and we were all set up load the little test patches into the software. We loaded the first one, which was a small little easy one that would only run if we had one of these reboots.

Well, we turned it back on and the computer started running just fine. We waited for a one more pass, and sent the commands to start generating image data and we started getting data, but we also started getting the resets. So now we are in the process of trying to get the software to skip selected parts of itself. It’s kind of like trying to remove your own appendix.

So now it’s the 17th, and we think we have a pretty good idea of what’s going on. We are pretty sure we’ve traced the problem back into a single electronic component in our computer. The good news is that we can tell the software to ignore the signals coming from that component, and just keep on running. The bad news is that every time the signal comes in, it’s possible that the software will skip one instruction. For some reason we haven’t figured out, the errors always occur at one of six particular instructions. So I had to look at what the software is doing just at that point and decide what the worst thing that could happen would be if those six instructions don’t work properly. So far it looks like it would have no permanent impact, just maybe a brief upset in the output data. So I’ve coded up a final patch to get MODIS back into full operational mode, and we’ll be trying it out over the next day or so.

And so once again, I can report that we are making slow but steady progress. It’s been a busy two weeks and there’s still more to do, like writing up a report on everything we’ve seen, tried, and speculated on for the last two weeks. Our basic mode of operation is to try things all day long, (which is well into the night for the guys back at NASA on the east coast) and think of something to try the next day. Then the NASA folks go home, and I stay late creating software patches. Then the NASA folks get in early to do all the management and approvals on the new plan, and by about 10:00 AM our time, we’re ready to start again.

It makes for long days filled with nervous excitement separated by periods of boredom and frustration. Still, one way or another, we’re going to get this puppy barking again. Hopefully before that poor guy taking measurements out in the desert in Africa gets tired of waiting for us.