digisth

If one is already well-versed in multiple areas of software technology (especially development and database administration), this is an excellent book. It surveys the landscape of software data storage technologies, talks about (at a modest level of depth) some of theory behind things like quorums in distributed database systems, resiliency/redundancy strategies during data loss, and a host of other interesting topics.

I'd consider its level of depth somewhere in the middle between specialist books and 10k foot overview books. I recommend it to anyone that has been a software developer or DBA for 5+ years, as I think they'd get the most value out of it.

If one prefers to use vim as a pager for psql, try this:

https://unencumberedbyfacts.com/2016/01/04/psql-vim-happy-fa...

I've been using it for a few months, and it works great for me.

These (and probably most) services are currently geared toward technology hubs and in-office work, that's true, as that's still the norm in industry, so they reflect that. I'm in NYC and saw plenty of outreach.

For remote, remoteok.io seemed pretty good when I used it.

The newer matching services (as opposed to boards) are all worth checking out: Hired, Vettery, Underdog.io. I found many good leads through all of them, and my current position is through one of them (Vettery.)

AngelList Jobs is also a place to find interesting positions (startup-centric ones in this case, as one might expect.)

Location: New York, NY

Remote: OK

Willing to relocate: No

Technologies: Python, Django, Ruby, Rails, AWS, Linux

Résumé/CV: http://www.panix.com/~sth/resume2017.docx

Email: spencer.hoffman@gmail.com

Interested in a backend role, especially web/data API building and/or data processing, broadly construed.

Do you know of a source that compares these different libraries in terms of capabilities, focus/use cases, size limits, performance, format support, etc.?

Googling turned up very little for me.

TIA

Edit: libraries mentioned in thread:

PMML, Arrow, Dill, marshmallow, pytables, parquet/fastparquet (and pickle, obviously)

The rule of thumb I've always used for when to use OO is "will there be more than one extant object at once or not?" If yes, and especially if these objects need real behavior, then use OO.

If you're essentially going through one object at a time, then discarding them, you're may just be doing conduit data processing, and so there's little advantage to using objects. I think what's missing in this (well-written) analysis is this distinction; if you're slurping data from one place, making a few changes (or especially if you're not making any), then sticking into a DB or vice versa, OO may be the wrong choice.

Ask yourself while writing the code: "are these active, behavior-driven objects that need encapsulation and relatively sophisticated behaviors, or is this just data I'm doing some relatively simple processing on?"

I have a pile of links for getting started with DL in my comment history you can use: https://news.ycombinator.com/item?id=10676455

What really helped advance my understanding from zero to knowledgeable novice was rewriting some existing code line by line (using expanded variable names and comments), and thinking about each line and what it does as you go. It's the software development equivalent of Hunter S. Thompson re-typing The Great Gatsby just to get the feel of writing a great novel. Here's one I did based on Denny Britz's tutorial:

Britz's Original: http://www.wildml.com/2015/09/implementing-a-neural-network-...

My version: https://gist.github.com/sthware/c47824c116e6a61a56d9

HTH

Mixins have their place, but perhaps we are at the point where we need general advice (like the class "Prefer composition over inheritance") for mixins ("Prefer traits over mixins"), especially as there's a good analog between the two. Matthijs Hollemans wrote an article which argues for that as well:

http://matthijshollemans.com/2015/07/22/mixins-and-traits-in...

The below resources are the ones I used when I started to learn about DL/NNs. Some of them are focused specifically on certain applications, but I found them helpful, too.

Basic NNs:

http://www.wildml.com/2015/09/implementing-a-neural-network-... (a whole series, all worth reading)

https://gist.github.com/sthware/c47824c116e6a61a56d9 (my code based on the above)

http://iamtrask.github.io/2015/07/27/python-network-part2/

http://rolisz.ro/2013/04/18/neural-networks-in-python/

ML:

http://cs229.stanford.edu/materials.html

http://onlinestatbook.com/2/index.html

DL in general, RNNs, RNTS, CNNs, some others:

http://cs224d.stanford.edu/syllabus.html

http://cs231n.stanford.edu/syllabus.html

http://mattmazur.com/2015/03/17/a-step-by-step-backpropagati...

http://arxiv.org/pdf/cs/0205070.pdf

http://colah.github.io/posts/2015-08-Understanding-LSTMs/

http://karpathy.github.io/2015/05/21/rnn-effectiveness/

http://nlp.stanford.edu/~socherr/EMNLP2013_RNTN.pdf

http://alexdavies.net/talks/

http://www.socher.org/index.php/Main/SemanticCompositionalit... roughRecursiveMatrix-VectorSpaces

http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/

http://colah.github.io/posts/2014-07-NLP-RNNs-Representation...

If you want to know more about RNNs in general, I can't recommend watching the videos/reading the notes from this course enough:

http://cs224d.stanford.edu/syllabus.html

If you want something more basic to get your head around NNs, I recommend Denny Britz's "Neural Networks from Scratch":

http://www.wildml.com/2015/09/implementing-a-neural-network-...

I created a gist with a heavily commented version of his code:

https://gist.github.com/sthware/c47824c116e6a61a56d9

I'm not aware of any efforts in that direction. As far as the review link, I don't think that would defeat the newer countermeasure involving 3rd party sellers shipping an empty box (which never gets returned.) It still looks like a verified purchase. Second-parties could require all items be shipped through them (meaning they receive the item and inspect it, then reship it to the customer), but the added expense seems like something they'd be unlikely to spring for.

It's not as easy as it seems. Many systems have already been devised, but fake reviewers adapt; it's an arms race, just like with email spam. Bing Liu, who has done work on sentiment analysis, has written about some techniques dealing with spam/fakes on review sites. Here's just a sampling of popular methods and flags:

- Duplicate checking (same user on different products with similar reviews, for example)

- Meta-reviews ("was this review helpful?" - can also be gamed)

- user rating averages (all highs or all lows sometimes considered a flag)

- ratio of "first product reviews" to total reviews on a per-product basis

- "Super-reviewer" status (questionable, but often a flag)

- Products with low sales ranks

- Review ring detection (IP block, post times, etc.)

- Early reviews

- Users who give high ranks, while most other reviews are low

- Positive reviews for one brand's products, and negative for others

- "Verified" purchases

A lot of these already have countermeasures, and fake reviews have already come up with counter-counter measures (like review-time staggering, using multiple IP blocks, shipping empty boxes to defeat verified purchases, etc.)

The "how to know what's trustworthy?" is the million dollar question, and it has many answers that happen to change over time. Not easy to solve.

Top of the list would be low-cost filtered API access. Much cheaper than the firehose, with commensurately less data (by keyword(s), location, etc. instead of the whole shebang.) The current pricing opacity and inability to just sign up online like you can with Parse (https://parse.com/plans) means far fewer people as customers. My understanding is that API deals are company-by-company.

A list of general suggestions for them: https://medium.com/@sthware/suggestions-for-twitter-hellowor...

Edit: I'll add that if relations get repaired now and then broken again, I don’t think they’ll be fixable a third time. They need to get this right, IMO.

Very interesting. Assuming that the drivers doing a "Rush" order can gather delivery items from multiple businesses in one geographical area, it could mean greater delivery efficiency (and shorter delivery times) overall. Example: orders come in to 4 businesses within within a few blocks of each other. Businesses send their Rush request in to Uber. Uber Rush driver indicates that they are in that area and picks up all items, then delivers them to target area(s).

Another thing this implies is that that these businesses could switch from a staffed delivery service to a completely on-demand one. One of the things the businesses and their customers could see an improvement with is the case where all their delivery people are out making deliveries, and new orders come in. With this, they just make a new Rush request and get a new delivery person.

They have/had the same problem with people with tattoos. Previous story:

http://gizmodo.com/the-fbi-is-struggling-to-hire-hackers-who...

There's definitely still a struggle in certain circles with the idea that many people in security (and other fields) are also members of subcultures that tend to have members that use recreational drugs, have tattoos, piercings, dyed hair, etc. Getting people to work for these agencies instead of the private sector is going to be challenging for a mound of other reasons; the least these agencies should do is rescind some of these bans.

There's nothing wrong with it, but having the ability for people to opt-out (some professional photographers, for example, don't want any kind of usage that isn't pre-approved, even if editorial) would be a way to make it crystal clear whether that user is OK with it or not. What we have right now leaves everyone in a murky area legally, and adds friction and annoyance. "Embed controls" would remove any questions about it, which would give users a say in how their content is used without having to give permission for each usage request, and would protect the people that want to use the posts without having to make a request for each one. Everyone would be covered.

Also, there are many different kinds of uses, some commercial-y. Here are a few:

- Display on Twitter itself (retweet)

- Use in a news story, near an ad

- Display in a Twitter widget on a web site

- Embedded Instagram photos on a company gallery page

- Use in an email ad campaign

- Use on a billboard

The list of things above exist on spectrum of "commercial-ness", and reasonable people can disagree on which they think are OK and which aren't as far as obtaining advanced permission. What many would like to see are much more explicit rules or technical controls so that the ambiguities are removed.

Embed controls might not be ideal from a fair use perspective, but it's a lot better than the "let the courts figure out" situation we have now.

Well, using the standard embedding tools, you do get attribution, and a link back to the source (see Twitter, Vine, and Instagram web embeds for examples), but the other parts are generally in the services' TOS. This is from Twitter:

"You retain your rights to any Content you submit, post or display on or through the Services. By submitting, posting or displaying Content on or through the Services, you grant us a worldwide, non-exclusive, royalty-free license (with the right to sublicense) to use, copy, reproduce, process, adapt, modify, publish, transmit, display and distribute such Content in any and all media or distribution methods (now known or later developed).

Tip: This license is you authorizing us to make your Tweets on the Twitter Services available to the rest of the world and to let others do the same."

Without this, all those embedded Tweets you see everywhere (like in news stories) wouldn't be available (this is why I think these services should all give you the ability to disable embedding your post(s) like Flickr/YouTube do. It would make everything much clearer and more straightforward.)

Joy Butler, an IP lawyer, seems to think that they would be covered by fair use:

http://www.guidethroughthelegaljungleblog.com/2014/07/copyri...

Hasn't been tested in court, though.

They are covered in the TOS/EULA, but based on some of things that have happened so far and some of the things written about it, it's far from clear that this stuff would hold up in court. You might think it should, and I think it should, but who knows if a judge would agree:

http://www.jdsupra.com/legalnews/an-update-on-the-legal-impl...

http://www.buzzfeed.com/jwherrman/want-to-publish-a-twitter-... -

https://pando.com/2013/01/22/how-twitters-new-embeds-will-ma...

Clear as mud.

Yeah, YouTube I knew about (though they do make the option a bit harder to find than they should) but AFAIK, there are no equivalents for Vine, Twitter, Instagram, etc. Facebook embeds I haven't looked at, though.

It's a complicated topic without nearly enough exploration in the legal area, and without any real agreement from the companies that run these social media platforms. Twitter's ToS (and their reps) have said you can embed anything without needing advanced permission; Instagram says the opposite. Media outlets use embedded tweets/Instagrams for editorial purposes everywhere, and this seems like it would be fair use (taking a snippet of a 140 character message as you would with an article makes no sense) but some people still complain. No one has any problem with embedding YouTube videos or Vines. How this all plays out is likely to be decided in the coming decades in courts and the media.

Three things that would be incredibly helpful in this area:

1) All these services should give users the ability to prevent embedding of their posts either globally or on a post by post basis. Drawing a very bright line like this makes what is and is not OK very clear. Off or on by default would be debatable.

2) Supply a UI or API for automatically obtaining licensing permission (requester goes to an Instagram post, clicks/taps "request license to use in/on {web, billboard, TV commercial, etc.}" which sends a message to the original poster, poster can than choose to consent/refuse, response is sent to the requester.

3) Make some public statements / legal guarantees (i.e., a promise to stand behind people who get sued even when using service-provided embedding tools) about what they consider acceptable usage.

Right now it's just very gray, and the safe route is to obtain advanced permission for anything with a photo, which I think goes against the frictionless spirit of the old web (i.e., just put up a link or excerpt a post and be done with it.) The service-provided embedding tools try to maintain that spirit in the more walled-gardeny world, but legal uncertainties still make that iffy. I wrote a bit more on this previously: https://medium.com/@sthware/user-generated-content-embedding...

I'm not sure what the legal reasoning would be for "hashtags as consent for photo usage", but something tells me that it probably wouldn't hold up in court if it came down to it.

Edit: another thing that needs discussion is things that seem innocent to many, but could be legally questionable. Prominent example: retweets. Let's say you embed a Twitter widget of your company's feed on your web site's product page. You retweet something someone has posted about your product. It then appears in the feed, on the web site, next to your products. Protected or not? Using some of the logic from Heigl vs Duane Reade, it might not be: https://social.ogilvy.com/celebrities-suing-brands-for-a-twe...

Considering that many smaller companies use e.g. CloudFlare's SSL, I'm not sure that would help them, but good to know.

I definitely agree with that part. Does your service check WHOIS data, then?

I really like this idea, but it seems fairly punishing for new / all remote / self-funded businesses. If you're running a company out of your own pocket (no SEC filings), aren't anywhere near big or old enough to have a BBB entry, and are remote (what street address should someone use? their apartment?) I'd like to know what signals you'd recommend bootstrappers use to engender trust for a service like yours, because I don't think "person is known on HN, posts on medium / Twitter about their service" is necessarily amenable to automated analysis.

I hope it is. At the very least, they should be trying. It certainly won't be an overnight process, but with time and enough developer-friendly decisions / public guarantees, it could happen. They should start yesterday, though.

As I've written in past comments (https://news.ycombinator.com/item?id=10094396), and as this post suggests, rebuilding developer relations and improving integrations would go a long way. There's a lot of potential locked in the platform right now; they should work on letting developers get access to it more easily and strive to remove the cloud of uncertainty that has built up around it (i.e., will they shut me down if I do something too popular that colors outside the current lines?) It would benefit everyone, especially Twitter.

It's certainly not required by a any means - if people are good and do their jobs well, then you don't need that. It might be harder to recruit people in some cases if they don't care about your product or service at all, though. There are some industries / technologies I don't have any interest in, and wouldn't work for companies involved in them. So there is some value in having employees care.

I think his point is one that seems obvious, but isn't always. First, you should believe in your product, then your job is to get others to do so. Treating people badly, not paying them, lying, etc. are all terrible and we should not ever praise those parts, but that doesn't mean his overall point is wrong at all. There are a few things going on there:

1) If you as the creator don't believe in it, you're probably going to have more trouble convincing others that it's worth using, unless you have some kind of interesting personality quirk.

2) The value of something - and it may be very valuable - might not immediately be obvious. AirBnB, for example, sounded insane to many people before it caught on. The founders had to believe in it, and they had to convince others to do so.

3) For things which are purely subjective (and I know some do find this distasteful, for sometimes good reasons), convincing others of its value is the entire point. Think about things like fashion, music styles, toys / novelties of various kinds - even certain kinds of technologies which only shuffle the deck in terms of functionality / UX may need to have users convinced. The effects of this one can be net positive (something starts out looking like a fad, turns into incredibly useful thing [Twitter]), net negative (unhealthy fad diets), or neutral (Webkinz? Furbies?), but even if you hate these things, you can't wish them away or deny they exist.

Some ideas: make public commitments to better developer-related policies that they promise to not renege on, outreach, reopen formerly closed APIs, lower-cost API plans so that incentives with small and medium-sized developers/companies are better aligned (developers pay something so Twitter doesn't feel like all value is being sucked out of it for free, less likely to shut things down.)