I'm finally joining the firehose club.
Introducing firehose.network - public infrastructure and tools for ATProto relays.
The initial release includes 3 public ATProto relays deployed globally: North America, Europe and Asia (new).
All are available for public use with a 72-hour backfill/replay window.
Check them out at:
I've been experimenting with indigo relay setup and maintenance for a few months now.
Setup is straightforward but daily operations are where it gets interesting.
A few things I'm running to keep things healthy:
PDS discovery
A cron job pulls the PDS host list from the main bsky.network relay every few hours, diffs it against firehose.network's relays, and adds anything missing.
PDS re-crawls
A script periodically checks for offline PDSes and sends requestCrawl to see if they've come back online.
Monitoring + Alerting
Email alerts for CPU, memory, disk space. Grafana dashboards for overall and per-relay metrics (alerting still WIP)
There's also a public status page.
(FYI, Asia relay is pretty new. Moved to a bigger dedicated server very recently)
What's next:
There are some early discussions on #relay-operators channel in ATProto Touchers Discord about making relay operations easier and have better insights, and a few tools are already built in this space.
My personal wishlist - either as a separate service / built in indigo itself
Auto PDS discovery + re-crawls
Expose per-PDS throttling metrics when event limits are exceeded (so I could setup alerts based on that)
PDS accounts browser in relay admin UI to easily help identify abuse (if any) and tune PDS limits
Still working on a few more tools to help both PDS admins and relay operators. More to come.
All these are hugely inspired by fig's() work on microcosm public infra and based on some suggestions from Bryan() during the ATProto_NYC meetup.
The relays are public and free to use. They already power a few of my own apps. Please do let me know if you use them and I can keep you in loop for future updates.
Follow for announcements.