Postmortem: Our first VLEO satellite mission (with imagery and flight data)

110 points by topherhaddad 4 hours ago on hackernews | 34 comments

[OP] topherhaddad | 4 hours ago

Founder/CEO of Albedo here. We published a detailed write-up of our first VLEO satellite mission (Clarity-1) — including imagery, what worked, what broke, and learnings we're taking forward. Happy to answer questions.

https://albedo.com/post/clarity-1-what-worked-and-where-we-g...

NoiseBert69 | 4 hours ago

Can you tell us some war stories about the software your group wrote for the satellite?

Stacks? Testing? Firmware Updates? Programming languages?

Thank you!

[OP] topherhaddad | 3 hours ago

Moving fast to make launch, we had missed a harness checkout step that would’ve caught a missing comms connection into an FPGA, and it was masked because our redundant comms channel made everything look nominal.

On orbit, we fixed it by pushing an FPGA update and adding software-level switching between the channels to prove the update applied and isolate the hardware path — which worked. Broader lesson, it is possible to design a sw stack capable of making updates to traditionally burned-in components.

roughly | 2 hours ago

> it was masked because our redundant comms channel made everything look nominal.

Hah, this has bitten me often enough I check for it in test suites now - ok, you’ve proven the system works and the backup works, have you proven the primary works? Another in the long list of ways you don’t expect a system to bite you until it does…

moffkalast | an hour ago

That's why starfleet always has a secondary backup /s

sjburt | 3 hours ago

The diffraction limit (under 1.22 h* lambda/d) of a 1m optic at 250km in visible light is about 17cm. How can you achieve 10cm resolution?

[OP] topherhaddad | 3 hours ago

Clarity is designed for a GSD (ground sample distance) of 10 cm. Generally the industry uses resolution<>GSD interchangeably. Agree it's not the true definition of resolution. But I'd argue the diffraction limit is an incomplete metric as well, like how spatial sampling is balanced with other MTF contributors (e.g. jitter/smear). For complete metrics, we like 1) NIIRS or 2) % contrast for a given object size on the ground (i.e. system MTF translated to ground units, not image-space units).

The main performance goal for us was NIIRS 7, and we decomposed GSD/MTF/SNR contributors optimized for affordability when we architected the system

s/learnings/lessons/g

prewett | an hour ago

At least it wasn't "learns", like "we had five learns from the project". Like, say, "ad spend". There's already a noun form of a verb, it's called a gerund: "ad spending".

Alasater | 24 minutes ago

From my perspective, the number one reason we had a well functioning satellite out of the gate is my philosophy of testing "safe mode first". What that means is in a graduated fashion, test that the hardware and software together can always get you into a safe mode - which is usually power positive, attitude stable, and communicative. So our software integration flows hit this mission thread over and over and over with each update. If we shipped a new software feature, make sure you got to safe mode. If we found a bug that prevented it, it's the first thing to triage. We build out our pipelines to simulate this as much as we could and then ran it again on the development hardware and eventually would load a release onto flight once we were confident this was always solid. If you're going to develop for space, start here.

sbierwagen | an hour ago

>The drag coefficient was the headline: 12% better than our design target.

Is the drag much better than a regular cubesat? It doesn't look tremendously aerodynamic. From the description I was kind of expecting a design that minimized frontal area.

>Additional surface treatments will improve drag coefficient further.

Is surface drag that much of a contributor at orbital velocity?

[OP] topherhaddad | an hour ago

Ultimately it's about the ballistic coefficient. You want high mass, low cross-sectional area, and low drag coefficient (Cd). With propulsion for station-keeping, it's challenging to capture the VLEO benefits with a regular cubesat. That said, there are VLEO architectures different than Clarity that make sense for other mission areas.

Yes it's a big contributor. The atmosphere in VLEO behaves as free molecular flow instead of a continuous fluid.

notaurus | an hour ago

How did you manage meaningful attitude control with only torque rods? They would need to big (read: heavy) to be useful — was this just stabilising in inertial frame or active pointing? Mag dipoles in chassis and components tend to lock tumbling satellites into the Earth’s magnetic field. Did you see this? Or did you see atmospheric drag dominate at this altitude?

Alasater | 58 minutes ago

I'm AyJay, Topher's co-founder and Albedo's CTO. We'll actually be publishing a paper here in a few weeks detailing how we got 3-axis torque rod control so you can get the real nitty gritty details then.

We got here after stacking quite a few capabilities we'd developed on top of one another and realizing we were beginning to see behavior we should be able to wrap up into a viable control strategy.

Traditional approaches to torque rod control rely on convergence over long time horizons spanning many orbits, but this artificially restricts the control objectives that can be accomplished. Our momentum control method reduced convergence time by incorporating both current and future magnetic field estimates into a special built Lyapunov-based control law we'd be perfecting for VLEO. By the time the issue popped up, we already had a lot of the ingredients needed and were able to get our algorithms to control within an orbit or two of initialization and then were able to stay coarsely stable for most inertial ECI attitudes albeit with wide pointing error bars as stated in the article. For what we needed though, it was perfect.

relaxing | 3 hours ago

So the root cause was the lubricant in the gyros couldn’t stand up to operating temperatures.

I’d be interested to read a postmortem of the systems engineering approach there.

[OP] topherhaddad | 3 hours ago

The lesson there - dig multiple levels deep in supply chain

Alas.. the speed & resources of a startup. But we're learning.

Neywiny | an hour ago

I work near space but not in space. I'm not sure I understand your process here. I see 2 possibilities: 1. You bought something the manufacturer spec lied about. While true we often validate specs, our terrestrial stuff is a lot cheaper so we can afford the spares. That said, if we buy something that doesn't meet the spec, you best believe we're taking the actions necessary. 2. This was built or designed inhouse, and the requirements didn't flow down correctly. That's also not great.

To be honest, postmortems (especially from startups) toe a fine line of scaring off investors, and this write-up seems a bit too glaze-y. I'm very happy for you that so much worked so effortlessly post launch, but that's more a success story than a postmortem. I'd like to see more of the root cause analysis for the issue, both technically and programmatically.

Alasater | 28 minutes ago

To be certain, if you're in the trenches of this anomaly investigation you'll get the full root cause and corrective action presentation, but that's not what this post is for.

You're correct on 1, we ended up hitting an edge case in their spec that they hadn't adequately tested to and the upper level management and engineering leadership were swift to accept the fault and implement fixes with us going forward.

From a SE perspective, as a "COTS" product, we had spec'd correctly to them, they accepted our requirements and then executed each unit's acceptance test plan (aka lower level than first unit quals or life tests where this should have been caught) on the ground without anything amiss. We ran through our nominal and off nominal cases at the higher level of assembly, but not for a duration that caught this on the ground. It wasn't until we were at extended operation on orbit the issues began.

Sadly like you state, space isn't like on the ground, you can't buy spares or replace things that fault, even for a true high volume COTS product that might slip through the acceptance testing.

wavesplash | 3 hours ago

Terrific writeup. Massive congrats to the whole team for all that creative thinking in flight and all that was achieved. (Add a note about updating FPGA's in space!) Looking forward to team Bedo unlocking VLEO for everyone.

heyflyguy | 2 hours ago

With image resolution this high, ground accuracy becomes an important factor as many people that prefer higher resolutions also want geospatially accurate images. Did you have any findings or results on this?

Alasater | 42 minutes ago

We actually didn't get to that part of the payload calibration campaign unfortunately, but all indications pointed towards getting geolocation between 5-10 meters on this first mission, driven primarily by star tracker quaternion error. Ephemeris and field angle map error was right in spec, so we were prepped to do an iterative line of sight pointing calibration but with the CMGs down, we didn't get to get there.

Future systems we've got a few updates though based on learnings, and we'll be shooting for closer to 3-5 meter geolocation error without ground control points (GCPs)

jacquesm | 2 hours ago

What an impressive project.

> Next up was maneuvering from our LEO drop-off altitude down to VLEO, where it would be safe to eject the telescope contamination cover

Why would it be unsafe to do this earlier?

> We had been tracking intermittent memory issues in our TT&C radio throughout the mission, working around them as they appeared. Our best theory is that one of these issues escalated in a way that corrupted onboard memory and is preventing reboots. We've tried several recovery approaches. So far, none have worked, and the likelihood of recovery looks low at this point.

Seems to be a pretty big problem as well, I wonder what their ideas are to diagnose the root cause here.

It all sounds a bit overoptimistic, but that may just be my interpretation.

MobiusHorizons | 2 hours ago

I think they want to get low enough that the jettisoned cover will not stay in orbit long or run into anything.

jacquesm | an hour ago

Ah, safe for others, not for the telescope. Thanks, I did not consider that option.

[OP] topherhaddad | an hour ago

Yes, exactly. Space safety :)

Alasater | 36 minutes ago

Space safety for sure on the cover, although I'm not sure we'll have that cover for future launches because it was less than easy to coordinate with the FCC on where to eject it.

The radio came from a supplier who has been investigating the issue. We had concerns with their NAND and ECC implementation, and we weren’t able to fully root-cause it with them. Going forward, we’ll be building our own radios, which will make it easier to test, iterate, and resolve issues like this internally, or at least be able to trace possible latch ups or destructive failures and implement the right levels of redundancy.

parsimo2010 | 2 hours ago

Congrats on having a successful mission, it seems quite successful for a first try, and you clearly have some talented people on your team. But I’m going to give you my unsolicited opinion on the writing style.

The writing style sounds more like a tech bro describing some weekend conquest, and is wholly unappealing to most of the space industry (or at least the ones with decision making authority). Your CMGs were “locked in,” several times you “nailed it,” and so on.

You might have a business strategy that I’m not aware of but I’d expect that most of your market is controlled by aging men in suits, and they don’t talk like this. Most startups and tech bros aren’t spending money on space. It’s big established corporations that can fund this kind of stuff. Write like them. You can talk like a tech bro and get seed funding, but if you want to get to a sustainable business you have to talk corporate.

I would hate for your company to get passed over for lucrative opportunities because your public image seems immature. I looked at your website and you have a bunch of ex-government people on your senior advisory board. Get their opinion on your writing. It sounds silly, but you significantly lower your probability of winning contracts if people see you as a team of “bros.” People don’t want to spend millions on guys who are “locked in.” People want to spend millions on people who do professional engineering and risk reduction and clearly communicate how professional and competent they are.

I ranted way too long about your writing style. It’s pretty cool that you were able to design your own bus and most of it worked.

metalman | an hour ago

it looks like Toppher Haddad has two writing styles, the tech bro blog , and the style he used above to discuss the technical aspects of there optical technolgy, and that is high geek, high uber geek of the type generaly associated with an inability to comprehend others lack of incomprehension, so actualy an unusual integration of skills and aproaches

Aurornis | an hour ago

> The writing style sounds more like a tech bro describing some weekend conquest, and is wholly unappealing to most of the space industry (or at least the ones with decision making authority). Your CMGs were “locked in,” several times you “nailed it,” and so on.

This is on the company blog. It ends with a call to action to either subscribe to their mailing list or explore their careers page.

It has the right tone for the goal. Tone policing isn’t helpful. The authors are even here answering questions which is very nice of them.

The whole thing gave off a feeling that the author was desperate to prove something.

wferrell | 32 minutes ago

Do you plan to offer an image service?

Great write up!

testing43523 | 30 minutes ago

What is the purpose of VLEO? Being at lower altitude to enable kinetic weapons for projects like Golden Dome?

VoidWhisperer | 17 minutes ago

I think the main purpose, atleast in this case, is to enable very high resolution satellite imagery (whether or not that being a good thing in and of itself is another matter).
Why yet another proprietary space electronics communication bus? Do we still not have a standard, useful, open space electronics communication bus? Can someone explain to me why not?