Google Domains blocking all Gitbook URLS: post-mortem

https://blog.gitbook.com/tech/post-mortems/06-20-gitbook-domains-blocked-by-registrar

by martypitt • 6 years ago

369 179 6 years ago

blog.gitbook.com

SamyPesse6• 6 years ago

GitBook CTO here:

Our production domains (gitbook.com and gitbook.io) have been blocked and locked by our registrar (Google Domains).

None of our infrastructure is impacted, all user content and databases are safe; our domains simply blocked by a heavy handed policy.

As mentioned on Twitter, we are all hands working with Google to fix this issues ASAP. We'll then share an in-depth post-mortem

https://twitter.com/GitBookStatus/status/1268554857411227648

lol768• 6 years ago

Seems like it's sorted now? It resolves for me

    $ dig gitbook.com a @8.8.8.8 +short
    104.18.9.111
    104.18.8.111

SamyPesse6• 6 years ago

Yes, after 6h without getting much responses from Google Domains support, we just got a notification that they unblocked our domains.

We are working on making sure that everything is correctly working.

Workaround that we've setup to allow our users to still access the platform through different hostnames will continue working.

catsdanxe• 6 years ago

If you wanted good customer support you shouldn't have gone with Google. There are plenty of other more reputable domain regrestrars.

CameronNemo2• 6 years ago

Do you have examples?

yoran• 6 years ago

I don't understand why you got downvoted. Google's customer support is notoriously non-existent (perhaps except for stuff that brings in money like AdWords). They admit themselves that it's a business decision: https://www.seroundtable.com/google-support-staff-limits-139...

MattGaiser3• 6 years ago

99% of people never need customer support, so I doubt most people consider it when choosing a service.

Operyl• 6 years ago

Is it related to the countless phishing pages hosted on your service? I’ve noticed an uptick.

antoineMoPa• 6 years ago

This is unrelated, but you are the founder of codebox.io right? I always wondered why the service disappeared.

SamyPesse6• 6 years ago

Yes, we've pivoted a few years ago to GitBook. Codebox was not working very well.

talideon• 6 years ago

As a former domain registrar, I would get the authcode, unlock the domains, and transfer them away as soon as possible. It's been a while since I read the RAA (https://www.icann.org/resources/pages/approved-with-specs-20...), but it's rather extraordinary to put a domain on clientHold, which is what I assume they did to you, outside of non-payment or some kind of legal dispute.

_jal3• 6 years ago

Srsly. I would love to hear about what Google told you.

...I don't host any DNS at Google, and if this is actually not spin, now I never would.

Not that I trusted Google before, but good lord, this is egregious.

Ayesh2• 6 years ago

Google is the registry of some novelty TLDs as well, and you are pretty much bound to dance the way Google wants with those TLDs.

SamyPesse6• 6 years ago

We've just published a postmortem: https://blog.gitbook.com/tech/post-mortems/06-20-gitbook-dom...

let us know if you have any questions!

dang• 6 years ago

Ok, we'll change the URL from https://twitter.com/GitBookStatus/status/1268528465990619137 to that.

I know it moves the rug under the existing discussion, but it's better than having two separate threads.

umvi• 6 years ago

Seems like you should also move off of Google Domains, unless you have some compelling reason to use them.

jtl9996• 6 years ago

Think you should investigate other options such as the known brand protection/domain asset management companies (MarkMonitor, CSC, easyDNS or their European equivalents)

EDIT: I see you're moving to Cloudflare, but I wish you the best of luck

cirno3• 6 years ago

I'm curious how you feel about CloudFlare as a registrar not allowing GitBook to use an external root nameserver.

Being forcibly stuck on CloudFlare's own nameservers only sounds very nefarious, and isn't a limitation I've ever heard of with any other registrar. For instance, it would break my tooling that uses my host's APIs to control DNS records through their nameserver.

I'd be very appreciative if eastdakota or jgrahamc could elaborate on what possible reasoning there is for this restriction as well.

mmglr• 6 years ago

How did you arrive at choosing Cloudflare? It's clear Google Domains has broken processes not conducive to running a business centered on user content. How do you know Cloudflare does not suffer from similar broken processes?

chris_wot• 6 years ago

Ever since you called my content "legacy", I've been leary of your platform.

azizur• 6 years ago

Do you use SPF and DMARC for your domain? You could have received instant reports to take actions sooner.

koluna• 6 years ago

Google Domains locked my domains at renewal time and refused to renew them until I provided proof of identity in the form of a scanned government ID -AND- a scanned copy of the credit card.

Coupled with the horror stories of non-existent support, the first thing I did was move my domains out this month.

CameronNemo2• 6 years ago

To where?

sbarre• 6 years ago

Not OP but I've been with Hover for 10+ years and their customer service is excellent.

bhhaskin• 6 years ago

I use Dynadot and haven't had any issues.

bigiain4• 6 years ago

I've kinda given up, and use Route53 for everything, on the grounds that if I managed to piss AWS off enough for them too take my domain names down, all my infrastructure will probably vanish to, so having someone else manage to keep my zone files up will not be of any use anyway...

It's got all my eggs in one basket - but solving that problem is a much bigger task that just picking a different domain registrar... (And I sometimes wonder if my blue-sky cloud-agnostic dreams would all come crumbling down if I even managed to implement then anyway - if whatever went wrong that pissed AWS off enough for them to shut me down got shared with and/or triggered the same reaction at Azure/Google/DO/whoever...)

rswail• 6 years ago

I use name.com for my domains, cloudns.net for DNS, fastmail.com for mail/calendar/contacts.

Currently don't have any hosting requirements but if/when I do, I either use AWS or digitalocean.

strooper• 6 years ago

Just out of curiosity- when Google is infamous for hard-to-reach human support, what in the Internet would make anyone interested to register their domain with them? Do they provide some sort of security or insurance that I am unaware of?

All the popular dedicated domain registrars I have used so far have excellent human support. Godaddy, namecheap, namesilo to name a few. I don't know if big companies or corporate use something more to secure their domain names and DNS, do they?

TeMPOraL• 6 years ago

Just out of curiosity- when Google is infamous for hard-to-reach human support, what in the Internet would make anyone interested to register their domain with them?

I think that, despite what frequenting HN may make you think, most people using Google's services - even the more complex or paid ones - aren't aware of the problems with support.

numpad02• 6 years ago

Living in an autocracy is perfectly comfortable up until very moment it no longer is

drusepth• 6 years ago

I moved/consolidated from GoDaddy and 101domain to Google Domains because of the support I've gotten from Google in the past (on Nexus/Pixel devices, Apps, Fiber, Fi, Stadia, etc).

I always assume the people complaining about nonexistent support from Google are trying to get support for something they aren't paying for. You pay for Domains, and the support reflects that. You probably can't get support for getting locked out of a consumer Gmail account or help uploading a YouTube video.

dannyw4• 6 years ago

As a paying G Suite customer, all of my support experiences have been horrendous, including trying to unlock an employee’s account that was locked for “spam”.

The support agent couldn’t do a single thing but tell me to wait for the possibly robotic appeals process.

coffeefirst• 6 years ago

Because I bought it 10 years ago, when Google was a very different organization, and moving would appear to risk some email downtime.

jamieweb2• 6 years ago

You can transfer a domain with no downtime. The name servers will remain during the transfer, so as long as your DNS isn't dependent on the domain registrar, you'll be fine.

If you are using the registrar's built-in DNS hosting, move away from this first, which can also be done with no downtime.

walrus012• 6 years ago

In terms of being able to open a ticket and get a response from a real human, for .ca domains the registrar canspace is quite responsive.

spacephysics• 6 years ago

After hearing this I’ll be transferring away from Google domains. Had to use them for the initial .dev sales

baryphonic• 6 years ago

Same. I bought a few .app names when they came out, but I'm transferring the rest away. It's not even the threat of someone accidentally locking something that concerns me, but the fact that Google is now so big that, like a black hole, all information is pulled in and none can escape, including (effectively nonexistent or incompetent) support.

thinkmassive• 6 years ago

"Everything is back to normal! The domains have been unblocked by the registrar. We are monitoring everything."

https://twitter.com/GitBookStatus/status/1268565887256330241

machbio• 6 years ago

The one surprising aspect about google domains - it is not integrated into Google cloud.. they are two separate products while AWS domain registration is part of aws cloud

bduerst• 6 years ago

Unlike Amazon, Google has more domain-related services than just Cloud - i.e. Analytics, Sites, Places, etc - which is probably why it's still independent.

gramakri• 6 years ago

It has no API as well

hahadeservedit• 6 years ago

I told you guys, don't use Google Cloud or any of their services, but GPC users thought they are smarter... keep shooting in your own foot.

robk• 6 years ago

If you were hosting phishing sites then I'm glad they did this. You should have better controls.

acituan3• 6 years ago

According to the postmortem they had removed the site in question a week back. Google was acting on a week old data.

Besides, any user-content serving platform will have to deal with malicious users. It is not a perfect process, especially against bot traffic. Shutting down the whole domain was very heavy handed.

mthoms• 6 years ago

I'll say it was. I can't believe they shut them down for content that had been removed over a week prior.

I realize some things take time, but the onus is on them to do a final verification before flipping the switch to shut down an entire business(!)

Google is "saving money" on support costs while simultaneously destroying their brand image (at least among the tech crowd). It will be near impossible for them to re-earn that goodwill. It's such ridiculously short sighted thinking.

I'm moving off of G-Suite this week. This is the final straw.

MattGaiser3• 6 years ago

There is spam on Facebook, there is spam on Reddit, there is spam on Twitter, there is spam on Hacker News.

Proactive spam control is not a solved problem anywhere. Banning a website over it is absurd.

ben5092• 6 years ago

Nuking a domain with no warning for any reason is nuts. They started to shut it down before the email notification.

And any time support or PR is droning on about "muh policies" you know there are problems.

heipei2• 6 years ago

Automatically detecting phishing sites is surprisingly hard to do reliably. The only thing you can really do is rely on flagging and human verification.

I've seen domains like google5[.]$tld which contained a fake Google Login form, up for close to an hour with zero detects in VirusTotal and no detection by Google Safe Browsing itself. If Google fails at detecting phishing against their own brand, what chances do smaller shops realistically have? Here's the example I mentioned: https://twitter.com/urlscanio/status/1178043405529763841

ben5092• 6 years ago

Is it even possible? If I'm phishing and I get shut down, I'd just modify my page slightly until it passes the detection.

It seems like a problem as hard as detecting spam.

Animats2• 6 years ago

The biggest hoster of phishing sites is Google.[1] Here's a list of major sites which have live entries in PhishTank.[2] Hosting phishing sites on Google Drive is very popular.

Many of those are long gone, but PhishTank hasn't cleaned them out, so they're still listed.

[1] http://www.sitetruth.com/reports/phishes.html [2] http://www.sitetruth.com/fcgi/ratingdetails.fcgi?details=tru...

heipei2• 6 years ago

To be fair, every platform which allows user-generated HTML pages suffers massively from phishing and most of them don't deal with it very well: Google, Microsoft, smaller players like Codebox and countless others. Then there's phishing on Dropbox, phishing on Google Forms, OneDrive, etc. Then you have phishing at all the various hosters like DigitalOcean, CloudFlare, etc. Even there you'll sometimes have IPs which have hosting phishing pages for various brands for a long time. It's not an isolated problem. Some deal with it more aggressively, true, but the pace and ease with which phishing can be stood up and modified makes it a whac-a-mole. Plus, the expectation is that most phishing pages will only be active for a few hours before being taken down and/or detected, so phishers pump out new ones on a constant basis.

I run the service at https://urlscan.io which tracks phishing and frequently run into these cases which render any kind of black/whitelisting impossible. Imagine Microsoft phishing hosted on Microsoft domains and infrastructure. Here's a fun search which will return lots of phishing on windows[.]net and googleapis[.]com: https://urlscan.io/search/#page.domain%3A(googleapis.com%20O...

Animats2• 6 years ago

Google tends to be at the top of that list, though. It wasn't always. When I first started doing that, MSN was on top, usually followed by Yahoo. For a while, Google Sheets were being used for phishing. You can put HTML in a spreadsheet cell, apparently.

I used to contact nonprofits and small businesses which showed up on that list. Inevitably,they'd had a break-in. With some nagging, I could cut the size of the list in half.

jamieweb2• 6 years ago

I love your service, I use it every day. :)

I find that a lot of phishing sites these days are hosted on legitimate sites that have been backdoored, but are continuing to operate as normal.

The phishing page is hidden away in the /.well-known/ directory or another similar one, so the website owner is non-the-wiser.

bigiain4• 6 years ago

To be fair, every platform which allows user-generated HTML pages suffers massively from phishing

The big difference here is that all the other platforms struggling (and failing too often) to prevent themselves being used as phishing hosting - aren't then turning around and hypocritically taking other platform's entire internet presence offline for doing so.

Google here are acting in the roles of judge and executioner, while being the biggest offender of the same crime.

dundercoder• 6 years ago

This mirrors my experience with google support, even as a paying gsuite customer. I lost a YouTube channel and wasn’t given any option to restore it. Their advice was to just re upload all the content and forget about view counts and old links that would no longer work.

boromi2• 6 years ago

I'm going through issues with G Suite as an admin. They randomly blocked my account falsely accusing me of sending spam. I can't even access the help support team sicne I can't login to the system.

fouric• 6 years ago

This is completely insane - there's absolutely no legitimate reason for Google to lock you out of your account, even if you were sending spam emails. Disable your ability to send new emails, maybe. Lock you out of your email inbox, maybe. But completely prevent you from accessing your account, and therefore even appealing? Inexcusable.

m0xte• 6 years ago

G Suite is a shit show.

Had major problems with a trial. Firstly it would just redirect back to the admin console if I hit gmail. Support couldn’t fix it.

Also you can’t give anyone access to YouTube (!) for 60 days after signing up or paying $30 in credit. Which is no good because when you pay them £30 fuck all happens. And again support were useless.

Happy O365 customer now. And I’ve used support which was excellent.

throw_m239339• 6 years ago

Same, and don't get me started on Ad-sense, kicked out by Google who claimed I was clicking on my own ads?!? They owe me tons on money, they know it because they sent me a stupid voucher at my home address to spend it on double-click, LOL... it was 14 years ago.

Since then every-time I have the opportunity I strongly oppose any business I work with to move to any Google cloud service and I was pretty convincing so far in large corporations, and I will continue to do so.

coronadisaster• 6 years ago

A bit off topic but Google blocks me constantly from accessing my gmail account... is there anyway to disable the "suspicious activity" "protection"?

ggm• 6 years ago

Having just moved 3 personal domains -> into google domains you can imagine I find this quite concerning.

I wonder how the domain registry community at large feels about this? ICANN exists, domains are subject to a legal agreement with ICANN, and it has customer-protection concerns surely?

hn_throwaway_99• 6 years ago

Well, the real question is whether you have full control over the content posted on those domains. If you do, you should be fine, but the lesson is you should never host anything with user-generated content on Google Domains.

dannyw4• 6 years ago

I can tell you about my experiences with Namecheap when I used fake WHOIS info for my privacy and someone reported me.

Their support staff said that due to ICANN policy, they must have accurate WHOIS info, so they have changed my WHOIS information but also added WhoisGuard to the domain so the details are not visible unless someone gets a Panama court order.

My trust in them increased substantially.

mikedilger• 6 years ago

Could you imagine if "youtube.com" were blocked and locked by IANA/ICANN/? because of reports that somebody posted a video with misinformation and phishing content?

I don't expect google appreciates how devastating this, but that would change quite immediately if it happened to them.

walrus012• 6 years ago

It's almost as if, while the laws and policies and regulations are supposed to be the same for all, giant entities with huge amounts of money and power behind them are held to a different standard than everyone else. I can't say I'm surprised.

lmm• 6 years ago

There's a parade of apologists who will quickly pop up to say that a private company platform should not be held to any free speech standard since they're not a government, never mind how much power they wield in practice.

nitwit005• 6 years ago

sent to an e-mail address on a domain they had just suspended...

I remember logging into an Ad Exchange account and discovering Google was sending emails to it. No one at the company was aware of it.

downerending• 6 years ago

Since no one else has gone there, does this qualify as "book burning"?

Ayesh2• 6 years ago

I don't think it should be registrar's duty to shut domains down.

Gitbook appears to use CloudFlare to host DNS, and likely to use their CDN/proxy, which makes them the actual "host" of the content. If there is any phishing, they should be the ones responsible.

The post says even their email was down. I'm deciding to not use Google Domains anymore.

Karupan• 6 years ago

I have a simple rule when using Google services (including paid ones) - be ready to lose access without a moment's notice. Unless you are an enterprise customer or can reach out to Google employees when things go horribly wrong, you are at the mercy of their automated systems.

As other posters have pointed out, hope they have a plan to migrate their domains elsewhere.

joepie91_2• 6 years ago

For the record: if you don't want this to happen to your site, you should avoid Dynadot also.

Several years ago I ran into almost exactly the same issue with them after someone sent them a frivolous abuse report - they'd locked down my domain unannounced to the point that I couldn't even transfer it out, and it took a call from a journalist(!) inquiring about the suspension before they were willing to unlock it for transfer.

I've been using internet.bs for most of my domains since and haven't had any issues with them - they let me know when an abuse report comes in, and give me the time to handle it. There are probably other registrars that are fine too, but I don't have personal experience with them.

wolco• 6 years ago

Never use google as a registrar. Support someone else. The internet is bigger than google.

techntoke2• 6 years ago

Support who? Government has power to take down any domain, especially TLDs and registrars in the US. People should be supporting decentralized DNS and name servers if they want to be safe from takedowns.

cube00• 6 years ago

If the government is taking down a domain at least there's a human behind it.

dannyw4• 6 years ago

The pirate bay disagrees with you.

_jal3• 6 years ago

You know, this is interesting. Google is taking something close to an editorial stance here - it is clear that they take action against their customers based on content on their sites.

Some might consider that sites with controversial content who use Google as their registrar and not taken down are considered reviewed-and-OK by Google.

Others might start pressuring Google to expand the nature of content they find unacceptable.

And yet others might try pressuring them to expand their policing - who knows how many bad actors could be shut down with Gmail monitoring?

notriddle2• 6 years ago

Phishing is not "controversial." It's illegal, and it undermines the principle of informed consent that meaningful freedom requires. Even if Google isn't legally obligated to take down phishing domains, I refuse to be so anal as to complain about it. It is everybody's job to make the internet safe and usable.

Too bad Google does a terrible job of it. No way am I signing up for a Google Domain in the future.

dannyw4• 6 years ago

Yeah, there’s nothing wrong with taking down phishing sites. The problem is when you misapply the restriction and take down a whole domain of a platform.

_jal3• 6 years ago

The point is not whether it is reasonable to do so or not; I agree that phishing sites should go.

The point is that there is a large lobby out there for knocking different types of content offline, and they don't show much concern for whether their demands also seem reasonable.

Most registrars feel they should not be in the content policing or law enforcement business. Google volunteering to do it in one area that makes sense opens the door for pressure to do it in other areas that make less sense.

tzs2• 6 years ago

sent to an e-mail address on a domain they had just suspended...

I've always assumed that:

1. When creating your account at the registrar you should use an email address that is not at any domain you will be registering through or transferring to them,

2. The contact information you give for your WHOIS records (or if using a WHOIS privacy guard service, the contact information you give them to forward to) should use an email address at a different domain,

3. If you use a DNS provider other than your registrar the contact email address they have for you should not be a domain using them for DNS, and

4. Same for whatever hosts your email.

The general principle is that the contact method that a service provider will use to contact you if there is a problem with your account or service with them should not depend on the account being in good standing and the service working.

mbreese6• 6 years ago

Aside from the Google domain issue (which is obviously a problem) -- why on earth would GitBook be hosting user-generated content under their corporate gitbook.com domain?

The registrar issue is one problem here -- but I don't see this addressed in their post-mortem. I thought this was a well-known issue. You don't host user-generated content on the same domain that handles your corporate email. Because things like this can and do happen, so you want to make sure that your company isn't also down while you're busy fixing a problem with your customers.

I know the exposure risks are different, but isn't this is one of the reasons why Github moved hosting Github Pages content to github.io from their primary github.com site? Or why raw data is hosted from githubusercontent.com (in addition to mitigating cookie security issues).

The registrar was an issue that was largely out of their hands. But this was something that they could control. And I think it's something that is missing from their post-mortem. If they had split their domains, then while serving user-content was disrupted, accepting users to gitbook.com and their email (!?!?) would have still been working. Also, depending on if they had a 3rd domain for hosting their CDN, users that had custom domains would also have been protected.This goes along with the idea that you don't use email from your primary domain (cto@gitbook.com) to register your domain (gitbook.com). Or host your status page on the same infrastructure as your site.

If I were them, after migrating registrars, this would be the next engineering change I'd make.

(Maybe they do this, I don't know enough about GitBook to know. But based on the thread here, I don't think they do, or at least it isn't mentioned in their post-mortem that I saw.)

jtl9996• 6 years ago

Where do you draw the line? I mean github.com is the corporate email domain of the GitHub company, it also hosts repositories (user generated content)

fred256• 6 years ago

But not for any binary content as uploaded by the user, that's served from raw.githubusercontent.com.

mbreese6• 6 years ago

But it isn't... directly.

Github.com is their corporate site, hosts their application, and is how we all interact with repositories (via https or git://). But, the only data you get from that site has been processed through their application and sanitized.

The only way to get access to the raw user-generated data is through raw.githubusercontent.com or Github Pages which hosted on github.io. And the data from raw.githubusercontent.com has the MIME types set so that you don't get HTML rendered -- only the raw plaintext (I think).

So, in this specific example, if there was someone hosting a phishing site in a github account, it would have been active only through github.io, not the main github.com site. (You could have likely seen the source code from the main site, but it would not have actually generated an HTML form).

For historical views, here is the Github blog post that details the change (2013): https://github.blog/2013-04-05-new-github-pages-domain-githu...

waon• 6 years ago

Assumingly because GitBook didn't expect Google to be so reckless and tone-deaf.

anoncareer0212• 6 years ago

because tech culture permits you to blame Google and their support team instead of knowing basic domain operation practices

acituan3• 6 years ago

Does it really make it less of a problem if they had a separate domain for corporate and it was only the user content going down? Serving user content can as well be part of a service’s core product and registrar induced downtime is a big enough problem.

Building and running anti-abuse tech works in proportion to the money put in. Can a small player really compete with Google’s anti-abuse investments? No. Save for a couple of exceptions, this means Google could always find abusive user content in its clients’ domains that the clients themselves couldn’t catch. I am not suggesting small players should be able to freeride Google’s anti-abuse capabilities, but if leads to enforcement and downtime of this sort, there needs to be a better protocol for the client to be able to respond or it paints a very anti-competitive picture.

shkkmo• 6 years ago

Does it really make it less of a problem if they had a separate domain for corporate and it was only the user content going down? Serving user content can as well be part of a service’s core product and registrar induced downtime is a big enough problem.

Yes, it really does. Not losing your ability to receive email will help resolve the problem.

Serving user content can as well be part of a service’s core product and registrar induced downtime is a big enough problem.

If that is the case, then choosing a registrar that provides quality, timely and responsive support when dealing abusive content complaints should be on the top of your priorities.

Building and running anti-abuse tech works in proportion to the money put in. Can a small player really compete with Google’s anti-abuse investments?

In this case the small company did find the content first through their own anti-abuse systems.

I'm not sure how this become anti-competitive. Google may have a monopoly on many things, but DNS registry is definitely not one of them.

thayne• 6 years ago

I can say from experience that moving content to a different domain without breaking things for users can be atrociously difficult. So if you make a mistake in how you split domains in the beginning, it can be very expensive to rectify later. Can you imagine going to all your customers and saying "you know all those links you have to example.com? You're going to have to change all of them to point to example.org now."

They _should_ split their domains, and given this recent issue, it would make sense for them to prioritize that. But, well, easier said than done.

mbreese6• 6 years ago

I imagine it will be painful to split their domains. But, I was surprised to see that wasn't mentioned in their post-mortem. Even if they didn't plan on doing it, I think that it would have made sense to mention.

I imagine their main paying customers would start demanding this change though. They are probably using a custom domain, so whatever domain GitBook chooses to use should be irrelevant for them (in theory, but in practice, who knows...).

grey-area• 6 years ago

why on earth would GitBook be hosting user-generated content under their corporate gitbook.com domain?

You mean like news.ycombinator.com, nytimes.com, facebook.com or most blogs or forums ever created?

Also their entire business is serving user domains so if that goes down customers don't care about their marketing site, though the email vanishing would be annoying. What's to stop google doing this with any regular domain with user comments or content though?

I blame the registrar - having heard this horror story I'd never host with google domains.

otterley• 6 years ago

Facebook uses fb.com as their corporate domain.

kohtatsu• 6 years ago

It most often has to do with browser sandboxing. See CORS, cookies, etc. It was a security flaw that began being addressed circa 2010.

SamyPesse6• 6 years ago

We don't host user content under the gitbook.com, or at least we've stopped doing it a few years ago.

User content is stored under *.gitbook.io, similar to GitHub.

Google blocked all domains that contained "gitbook" in our account, even ones that are used for some infrastructure and are not accessible by the public. We don't know the exact reason for this, maybe they've blocked gitbook.com because we still have some redirect for content that was hosted under it years ago.

And yes we are going to make changes to host our status page under another domain.

mbreese6• 6 years ago

That’s really important to add to the story! I think that makes the rest of the story more troubling. Especially knowing that domains unrelated to the apparent user phishing code were affected.

You might think of adding it to the post-mortem doc so that others don’t assume what I did.

aledalgrande• 6 years ago

Wait until Google controls all of your traffic, with AMP, ads, Google Cloud and all their services. Flip the switch off for a business and kill.

johnghanks• 6 years ago

gitbook... who?

Keverw• 6 years ago

Makes me wonder if they don't understand user-generated content, if Google Domains is more meant for small personal sites maybe?

I used to use a hosting company that had in their terms, that profanity wasn't allowed... The company seemed to have decent hardware, user interfaces and US based support too but ran by Mormons based in the beautiful state of Utah. I don't have much of a opinion of Mormons but I think pushing religious views on your customers is bad business.

So after noticing that in their terms I was curious since I run a personal WordPress blog, if someone wrote a comment even if it wasn't approved just by being in the database if it broke their terms and was told yes... Now that I'm a bit older, I wonder if they really are scanning database table text fields for swearwords or that support rep misunderstood what I meant... But after that discussion, I switched my domain and hosting elsewhere, two separate companies actually instead of just one account with both domain and hosting. If some spam bot or troll writes the F word, they are going to take my entire site down even if no fault of my own? Yeah, no thanks.

They got acquired though by a lot larger company and doesn't seem to have those same terms now though, and heard they now outsource their support. There was someone I used to talk to who was a big fan of them though and referred me. So slightly over a decade later, they are probably not even remotely the same company anymore though since merged with a much larger company. Wouldn't surprise me if they ended up getting rid of a lot of support staff and just kept a handful of people to manage the datacenter, since marketing, accounting, support, etc would probably be centralized between all the hosting companies they own I'd imagine, which is probably a blow to the local areas when they merged unfortunately. I guess that's one of the sad things about getting bigger, they lose that small town friendly startup feel probably as seemed like a great company other than forcing religious views on people. Just did some more research about their company, sounds like they were also anti-lgbt too sadly. Makes me wonder how many other people working their share those views or just something management was pushing down since looks like the church changed their policy last year... I've been wanting to go somewhere with more opportunities, and actually Salt Lake City is one of the areas I've been considering since seems like a bit bigger city and on the list as a place for upcoming startups.

antoncohen• 6 years ago

Are they conflating domain registrars and DNS hosting? Did Google Domains block their domain at the registry level, like removing the NS records from roots? Or did they block is at the domain DNS level, like not resolving queries to Google nameservers for the domain?

I don't expect many startups or big companies use Google Domains for DNS hosting. AWS Route 53, Google Cloud DNS, or dedicated DNS hosting would be used for that. But I expect it is fairly common for startups to use Google Domains for domain registration.

rydre• 6 years ago

Try epik.com

CompuIves• 6 years ago

The exact same has happened with CodeSandbox, Google Domains had blocked our csb.dev (internal domain) and csb.app (domain for projects) without any warnings. It took us two days (of downtime!) to get them to lift the block. But then after 3 weeks, they did the exact same thing! They blocked us again for reports without warning. Luckily we had a fallback domain so that there was not much service disruption, but it was extremely frustrating working with them on this.

After 3 sudden blocks from them, they now give us a warning before they're planning to block us, but it took many calls before we got to that stage.

wegs• 6 years ago

... Which is again why I would never use Google for any business application. These stories happen all the time.

"I built my business on Google Cloud / Android / Google Apps / Youtube / etc. I was flagged by an automated system. There was no human to speak to (at least one competent or empowered to do anything), and my business is now gone."

Some of these go viral on social media, and Google then fixes them. Some don't.

Support channels built on only having an impact if things go viral aren't what I'll build my business on.

franga2000• 6 years ago

A month or so ago, I was setting up a new system for writing internal-ish documentation and made the rather unpopular decision to not go with Gitbook because it was too reliant on their servers for my taste. Today, I got to send a very satisfying "told you so" the the team chat.

To be clear, I think the Gitbook team actually handled this very well and it wasn't their fault, but it does illustrate very clearly why we shouldn't be relying on cloud services when we really don't need to.

Spivak• 6 years ago

I’m surprised how few people in this space aren’t packaging their apps up for on-prem customers.

* Fewer security and compliance hurdles when its on our own infra.

* Perfect uptime because it’s our responsibility. (Not saying we’re better but outages are our fault and don’t make you look bad)

* Fewer hurdles when integrating LDAP and friends. There’s no need for the weird “connectors” that companies come up with.

* We’ll pay you more for the privilege of hosting it ourselves.

* Giving us access to the code in a shared source model buys you an extreme amount of lock-in since once we start tweaking our fork for our needs we’re not leaving.

* The support burden is higher, sure but if it’s your most expensive enterprise tier you’ll almost always be working with a professional IT team.

* We’re the easiest crowd to up sell and we’ll even pay you to develop features you can sell to other people.

I don’t want to make it seem like selling in enterprise isn’t a slog but it’s a good business to be in.

kerng• 6 years ago

Using Google products really becomes a big liability for businesses. Articles like this have become quite frequent over the last year.

ezoe• 6 years ago

When google said "Jump", you must say "How high?". But there is more. Google won't inform you the required height or doesn't define what jump is.

Google Domains blocking all Gitbook URLS: post-mortem

Related Stories