zagi

Yesterday, network access was taken down to sharethesafety.org which was hosted on our service by a customer, an intermediate platform provider. We want to let you know what happened.

On June 23rd, we received notice from the NRA's counsel that sharethesafety.org was infringing NRA’s trademarks. The NRA demanded that we remedy the problem by removing the material. In response, we followed our standard procedures for trademark infringement notices and informed the customer immediately of the NRA’s demands. We provided more than 5 days for the customer to respond to or resolve the issue. We also spoke to the customer on several occasions to inform them that we intended to take action on the trademark claim. They chose not to remove the violating material. Accordingly, our Trust & Safety team restricted network access to their Droplet, which caused an outage to all of their user’s websites hosted on that Droplet. In less than 2 hours of the outage, the customer was able to address the trademark notice, and network access was immediately restored.

DigitalOcean followed procedures that help protect us from having to resolve what can be complicated disputes between third party rights holders and our customers regarding IP issues. In this case we should have given greater care to the customer’s voice and their right to engage in parody. In retrospect, we believe that the website identified in the NRA’s takedown notice was not a trademark infringement but was instead protected by the First Amendment. We at DigitalOcean champion freedom of speech and the free and open web.

Going forward, we will be working closely with our legal counsel to review our Trust & Safety procedures so we can make better decisions. We are committed to providing our customers with the best level of service and supporting their rights and freedoms. That is our responsibility as an infrastructure service provider and one that I take very seriously.

Sincerely,

Ben Uretsky Co-Founder & CEO

Hey, full disclosure I’m the CEO and cofounder at DO. We built DigitalOcean specifically for developers at startups, SaaS and small-medium businesses concentrating on simplifying cloud infrastructure and delivering an experience that people love. Today we offer a high performing compute product with Droplet that allows running nearly any Linux workload. We also offer shared networking, Floating IPs allowing DIY load balancing, and team accounts for collaborative groups. As well as one click snapshots, automated backups, a web console, extremely friendly support, and tons of documentation that make DO a perfect fit for startups.

Our differentiators allow devs to get started faster, leveraging our UI and intuitive API to automate their deployments and scale with ease. Some of our larger customers have started with just a few Droplets in a stage/dev environment and have grown to thousands of production Droplets managed via their application.

We believe our platform is ideal for startups coming to market that need flexibility and ease of use when going through rapid feature iterations.. Our philosophy is that developers should be focusing on their applications, not managing their servers. This is baked into everything we do, including our straightforward pricing model, so that there’s no need to guess what your bill will be at the end of the month.

Also our price:performance ratio is outstanding and with a static pricing model it's trivial to calculate your bill and plan ahead for growth. And we provide the same on demand utility per hour pricing that allows maximum flexibility. One click snapshots, automated backups, web console, extremely friendly support, tons of documentation make DO a perfect fit for startups.

If you’d like to consider DigitalOcean for your startup just reach out to us via https://www.digitalocean.com/company/contact/sales/ and we’ll support you.

Thanks for pointing this out, nearly 1 year ago we implemented a backup mechanism which stores a destroyed machine for 24 hours. This is only enabled for users with a valid paying account and we used this mechanism sometimes to return a droplet to a customer who accidentally deleted it, other times there were problems with an integrated third party where we were able to recover customer droplets because of a security problem.

We will take the necessary steps so that if users are enabling scrub all data permanently then we will not store this temporary image and therefore destroys will be immediate and permanent.

Hi, Ben from DigitalOcean here - just to give you guys an update. This method will no longer work on a newly created droplet.

We've now default scrub_data to ON for both web interface and API as we look at making this process permanent. Additionally, we've re-engineered the way we're provisioning disks and access to previously written data is no longer possible.

We've taken all steps in favor of security currently and will build a permanent solution that favors security and caution moving forward.

Hi, this is Ben, CEO and Co-Founder of DigitalOcean, we have received the document and will discuss the matter publicly.

-----

All times are UTC.

Our monitoring picked up a malicious UDP traffic pattern on 2013-09-08 00:58:23. A ticket was then opened with the customer at : 2013-09-08 01:05:55 roughly 7 minutes later.

The customer informed us that it was a script that was crawling in the background.

We informed the customer that it may be a good idea to check through the virtual server to see if there were any signs of a compromise just in case.

The droplet was unlocked at this time.

A second UDP pattern was detected on 2013-09-24 12:27:09 and a ticket was opened 2013-09-24 12:27:14 to request more information from the customer.

Because this was already a second occurrence we had to do a more thorough follow up. Discussing the matter with the customer, he informed us that it was a mysql db dump script that was pushing data to dropbox.

He provided us a link to a github project that he wrote, we asked further questions. Specifically if you are writing a mysql dump remotely why are the packets being sent as UDP? Additionally if the final destination is dropbox that would be an SSL encrypted connection and again why would that transfer go over UDP?

We reviewed the code of the dump-to-cloud project and it was using the dropbox sdk, here is where the file transfer is initiated:

   def upload_file(file_name)

        client = DropboxClient.new(@access_token)

        file = open(file_name)

        puts 'Uploading file!! Please wait.'

        response = client.put_file("/#{file_name}", file)

        puts "uploaded:", response.inspect

    end

   def build_url(url, params=nil, content_server=false) # :nodoc:

        port = 443

        host = content_server ? Dropbox::API_CONTENT_SERVER : Dropbox::API_SERVER

        versioned_url = "/#{Dropbox::API_VERSION}#{url}"



        target = URI::Generic.new("https", nil, host, port, nil, versioned_url, nil, nil, nil)



        #add a locale param if we have one

        #initialize a params object is we don't have one

        if @locale

            (params ||= {})['locale']=@locale

        end



        if params

            target.query = params.collect {|k,v|

                CGI.escape(k) + "=" + CGI.escape(v)

            }.join("&")

        end



        target.to_s

    end

    def do_put(url, headers=nil, body=nil)  # :nodoc:

        assert_authorized

        uri = URI.parse(url)

        do_http_with_body(uri, Net::HTTP::Put.new(uri.request_uri, headers), body)

    end

Given that it was the second incident that a UDP traffic pattern was observed in less than 30 days and that the information the customer provided regarding the traffic did not match up, we made a determination that in fact it couldn't be this script that was generating the traffic.

All of this information was relayed to the customer that we did not believe that the traffic in question was related to this script because it would not rely on UDP, an insecure protocol to deliver files to a secure endpoint where data integrity was of the utmost importance.

Unfortunately, we could not unlock the account at this time because the information we received was not clear and we already had two incidents of outbound UDP traffic that appeared to be disruptive and abusive in nature totaling 1Gbps as if it were a denial of service attack, typically associated with UDP packets.

A signed letter allowing us to discuss your private account details that would violate our 'Privacy Policy' and reveal the factors that went into making the decision.

All machines in NYC2 region are accessible via this private network.

That is correct, it's not bound by tenant.

Most other providers do not restrict private network either, I'm talking about the big ones like RackSpace, Amazon and others.

What you're talking about is dedicated private VLAN's or private subnet's and that is not common especially in a cloud environment.

A cloud is not defined by high availability. Just look at Amazon's cloud - so many single points of failure. Amazon says to distribute your application and redundancy across multiple redundancy zones.

Cloud is defined as:

1. On demand automated provisioning 2. API interface for programatic control 3. Infinite resources from the customer's perspective 4. Virtual addresses for physical location 5. Utility measured service

This is the definition from NIST : http://csrc.nist.gov/publications/nistpubs/800-145/SP800-145...

So in that context DigitalOcean and Amazon are both Cloud providers.

The default is 5 servers to allow us to verify users. If you hit the limit you can request more servers, there is no actual limit if your account is verified and in good standing.

Hi, I'm the CEO of DigitalOcean and we are not operating on a fantasy revenue model. All of our unit economics are positive and we are certainly here for the long haul with a sustainable business model.

To answer your question directly, and without knowing the exact cost structure of our competitors, I would have say that we are generating less margin per unit but we are overall sustainable and growing healthy.

We are using a Cisco based network and working closely with them to build a resilient network architecture. We've already encountered a number of bugs within Cisco's platform that have surfaced and are working closely with their TAC to escalate and resolve them permanently not only for DigitalOcean but all providers world wide.

@6. It is perfectly fine to use multicast ElasticSearch clusters on DigitalOcean - just make sure to use an ACL to protect the environment in a public cloud environment.

RAM is dedicated on DigitalOcean, CPU's are not - the virtual cores are shared to some extent on almost any cloud provider if you do the math.

Even though our pricing is low we still generate profit on every unit that we sell, therefore we will be around for a long time. We have a ton of expertise in the server industry and we've already been around for 18 months. Here's looking forward to the next 18 years :)

We do not oversubscribe memory and we do not provide SWAP space by default. Since its SSD based and memory is so cheap we recommend users size the machines according to their needs/requirements.

This is one of the fundamental differences between a platform and an infrastructure provider. As a platform you build a single tool that is used by everyone and with it come inherent limitations. If an infrastructure provider can give you the building blocks to create a platform service on your own, you maintain full control. So long as your time investment into growing the infrastructure doesn't outweigh the cost savings of a platform you get much more predictable growth, better pricing and maintain full control allowing you to develop a product that's customized to your needs exclusively.

DigitalOcean recently became fully integrated with Cloud66.com just last week. It's a great way to easily spin up a new server, and echo's our mantra of simple cloud hosting.

>My personal question is: are DigitalOcean making a profit?

Yes. If you do the math they actually do break even and manage to squeeze some profit.

It is one of our most up-voted features and we will begin building out storage solutions this month now that our SSD/RAM upgrades are finished.

We are still deciding whether to roll out first some sort of block/tiered NFS storage that can be mounted or something API driven like S3, so feedback is welcome.

We've seen a high failure rate on SSD compared to SAS & SATA drives. All of our servers are in a RAID environment, so a single failure does not take the system offline. Additionally our backups and snapshots provide another level of redundancy against disk failure. So far no data loss.

Thanks! We definitely got a surge in traffic from the TechCrunch post and have been able to stabilize the frontend. Everything should be working normally at this time. We'll continue to add resources to make sure we can handle user requests going forward.