Ask HN: Is GitHub down?

by mikebonnellon 6/29/2023, 5:42 PMwith 184 comments

Not loading for me at all, but status page shows green across the board.

by edgyquanton 6/29/2023, 5:53 PM

Everytime github goes down, and my push/pull is rejected, I immediately assume they’ve discovered I’m incompetent and fired me. And I’m the head of engineering at my company.

by klysmon 6/29/2023, 5:46 PM

Seems to be a fairly catastrophic failure. https://github.com/ fails to load. https://www.githubstatus.com/ shows all green as of this writing. Nothing on the twitter yet https://twitter.com/githubstatus

edit: The outage is now acknowledged on the status page https://www.githubstatus.com/

edit: EU folks appear to have things working so it looks like a regional network fault

by hn8305823on 6/29/2023, 5:49 PM

140.82.113.0/24 is visible in the global routing table:

  route-views>sh bgp 140.82.113.0
  BGP routing table entry for 140.82.113.0/24, version 62582026
  Paths: (19 available, best #4, table default)
The route is verified by RPKI so it's not a route hijack.

Edit: deleted traceroute

by mbreeseon 6/29/2023, 5:53 PM

https://www.githubstatus.com/

Just flipped to red.

See here: https://www.githubstatus.com/incidents/gqx5l06jjxhp

>Investigating - We are currently experiencing an outage of GitHub products and are investigating.

>Jun 29, 2023 - 17:52 UTC

by lucb1eon 6/29/2023, 5:59 PM

https://codeberg.org open source GitHub without Microsoft (it's a German non-profit). You can also host your own lightweight https://forgejo.org instance

In case anyone questions whether centralizing literally everything onto GitHub is a good idea, at least as a mirror for things you depend on

by stefanos82on 6/29/2023, 5:46 PM

So down right now...I wonder why they still use https://www.githubstatus.com/ that reports everything is alright when it's not!

by lucb1eon 6/29/2023, 6:03 PM

To all the "same" and "not for me" posters: the very least you could add is a location

by hejcloudon 6/29/2023, 6:04 PM

I feel sympathy for all the engs at companies I've implemented CI/CD based on Gh Actions in recent years. It's not like I didn't tell them that Github showed to be somewhat unreliable in the recent years and in contrast to their claim "it's just the build pipeline, not the product" I think it is a horrible incident if you're not able to deploy to production and have barely any ad-hoc backup.

I'm always evangelizing Argo or Flux and some self-hosted Gitlab or gitea, but seems like they all prefer to throw their money at Github as of now.

by ActualHackeron 6/29/2023, 6:02 PM

Since nobody can work, I'll just leave this here: "I must have put a decimal point in the wrong place or something. Shit! I always do that. I always mess up some mundane detail."

by charlieyu1on 6/29/2023, 5:47 PM

Loads for me

Unless it is cached

Edit: I could even login

by whitewingjekon 6/29/2023, 5:47 PM

Yes, github status says all is operational but hacker news is faster. Go figure.

by gaoshanon 6/29/2023, 6:03 PM

My team ran some code that crushed our Github actions very shortly before this outage. Nervous laughter around the office that it was us.

by sdsdon 6/29/2023, 6:07 PM

I am laughing so hard right now. My best friend mocked me for using git.lain.faith to host my code, saying it wasn't reliable. Well, well, well. In the last year GitLain hasn't gone down once.

I know he was still right in a way, who knows when git.lain.faith will just disappear. But still. I'm going to send some texts to bother him right now, hahaha.

by onionisafruiton 6/29/2023, 6:00 PM

I heard from somebody at GitHub that this one will make a good incident report. No other details or estimates for recovery time.

by Macuyikoon 6/29/2023, 6:07 PM

Update

We have identified the root cause of the outage and are working toward mitigation Posted 4 minutes ago. Jun 29, 2023 - 18:02 UTC

by oblongatacuron 6/29/2023, 6:03 PM

I noticed Github's OIDC token changed about a half an hour ago. Security incident?

by jestermanon 6/29/2023, 5:52 PM

From GitHub - Incident On 2023-06-29: https://www.githubstatus.com/incidents/gqx5l06jjxhp?u=ry1xb4...

by renonceon 6/29/2023, 6:02 PM

Not down for me accessing from Hong Kong. I suspect this is a regional outage.

by rvzon 6/29/2023, 6:26 PM

Unsurprising. There is at least one outage with GitHub every single month. [0]

[0] https://news.ycombinator.com/item?id=35967921

by turtleyachton 6/29/2023, 5:49 PM

Won't load for me.

https://githubstatus.com shows all green, but it's not the case...

by kashnoteon 6/29/2023, 5:49 PM

Are these status pages updated manually? At the very least it should be able to detect that the home page doesn't even load and turn itself red.

by gwbas1con 6/29/2023, 6:17 PM

Yaay! I just pushed and my new commit showed up in CI!

by gwbas1con 6/29/2023, 5:45 PM

I can't push from my desktop, and https://github.com/ spins in the browser

by computershiton 6/29/2023, 5:43 PM

Yup. Totally down, happened right as I opened a PR.

by acimon 6/29/2023, 6:28 PM

I just noticed that artifacts download didn't work although the web site was up. There was some varnish proxy error.

by numbsafarion 6/29/2023, 5:52 PM

If they used AI to rebuild their system and migrated it Azure, I bet they would stop having all the problems.

by joshstrangeon 6/29/2023, 5:45 PM

Yep, looks like it's down. Can't pull/push and can't even get the web to load at all.

by aftaylor2on 6/29/2023, 5:45 PM

Unable to load Github website or push using git-remote-https as of the last several minutes.

by MorningInfidelon 6/29/2023, 5:45 PM

Same. Spins for a while then dreaded `ERR_CONNECTION_TIMED_OUT` chrome error page.

by saintblasphemeron 6/29/2023, 6:11 PM

Looks like we are back online.

by distortionfieldon 6/29/2023, 5:52 PM

Down for me as well, Operation Timed Out errors on all attempted SSH connections

by Gordonjcpon 6/29/2023, 6:00 PM

No it isn't, it's working absolutely fine and has been all afternoon.

by ny711on 6/29/2023, 6:26 PM

This should be a weekly ASK HN; seems to happy pretty frequently at this point

by circuit10on 6/29/2023, 6:01 PM

It works for me at the moment

by JohnMakinon 6/29/2023, 5:49 PM

First noticed when trying to pull a helm chart - get a 503 backend error page.

by squaloon 6/29/2023, 6:51 PM

Not a single day passes without a MAJOR outage in a Microsoft owned service.

by escape-big-techon 6/29/2023, 6:11 PM

often people point out how unreliable self-hosted services are, well, hosted services are just as unreliable if not more.

this, ladies and gentlemen, is why you should always self host critical infrastructure

by coder9874on 6/29/2023, 5:47 PM

Yep it's down. Why do they even bother with the status page

by connorgutmanon 6/29/2023, 6:06 PM

Up in Africa (Morocco).

by dataluson 6/29/2023, 6:02 PM

And just when I was about to get into flow state...

by duckkg5on 6/29/2023, 5:53 PM

Status page updated showing all red https://www.githubstatus.com/

by GrumpyNlon 6/29/2023, 5:45 PM

Same here, it cant send a verification sms.

by jetpackjoeon 6/29/2023, 5:47 PM

Its down for me

by ukrainiancrabon 6/29/2023, 6:04 PM

Cannot use oauth2 in algoexpert :/

by wcarsson 6/29/2023, 5:45 PM

I can't even load the status page.

by Xeamekon 6/29/2023, 6:03 PM

looking on the bright side, at least we'll get an interesting post-mortem to read in a day or two.

by ushakovon 6/29/2023, 5:47 PM

Actions won't start for me

by mydriasison 6/29/2023, 6:03 PM

Yes. Can't even pull ;(

by ughitsaaronon 6/29/2023, 6:28 PM

Good time to grab a beer!

by tednotfredon 6/29/2023, 5:48 PM

Down for me right now.

by sparc24on 6/29/2023, 6:05 PM

Azure Strikes Again!

by themusicgod1on 6/29/2023, 6:31 PM

It shouldn't matter. Nobody should be using github post-2018.

by facu17yon 6/29/2023, 5:59 PM

seems to be a network issue, not a service issue

by almost_usualon 6/29/2023, 5:50 PM

Works for me?

by etimbergon 6/29/2023, 5:43 PM

Same for me

by abe-101on 6/29/2023, 5:49 PM

Same here

by ActualHackeron 6/29/2023, 5:46 PM

Yep

by saintblasphemeron 6/29/2023, 5:44 PM

Same here.

by rk32on 6/29/2023, 6:01 PM

its all down currently

by adpirzon 6/29/2023, 5:43 PM

Same