carljv

Mazda is pretty allergic to touchscreens and views them as a safety risk. Environmental controls are physical button and knobs in all models I’ve seen, and even the infotainment feature are controlled with buttons and a central knob. I never have to reach up and touch the screen. The screen isn’t even responsive to touch while the car is in motion.

First, the link should probably be to the full report

https://www.kaggle.com/nomilk/data-science-language-and-job-...

Second, this is data scraped from Australian job listings. So the title should probably reflect that.

Lastly, this data seems a little odd. You’ve got month-to-month swings of 20% or more happening here. This data source seems extremely noisy and I don’t think there’s anything you can reliably say about a trend here.

I think others in the comments have pointed out why, if it were real, you might see this overall trend and why it doesn’t mean salaries for the same role are falling.

This looks like a fun side project, but I would be careful reading anything into the results.

In addition to that, I think some lower-paying roles that used to be analysts or BI specialists are increasingly being called Data Scientists. My guess would be if you controlled for skills and job responsibilities you’d see increases.

That is not an accurate description of the book.

There’s some overlap, but vectors are essential to the language. Every type of data in R is a vector. There are no scalars, just vectors of length 1. Instead of dictionaries, it’s idiomatic in R to use “lists”, which are vectors of vectors. Data frames are lists (vectors of vectors) constrained to have equal length element vectors (ie columns). Classes are defined as lists with some metadata (stored in a vector) to direct method dispatch.

It’s not just vectorizing mathematical operations a la numpy.

This. During the period where Uber and Lyft weren't operating in Austin, there were several local alternatives that popped up right away, worked just fine, operated within local regs, and that I honestly preferred service-wise. If Austin can do it, I bet California can.

If one is working on AI dealing with facial recognition and is oblivious to the potential for bias, and the unethical applications of that technology, at this point in the game, I can only assume it's willful.

It's not the job of the AI researcher to solve "social biases" in every field, it's their job to build the AI.

"'Once the rockets are up, who cares where they come down? That's not my department' says Wernher von Braun"

It’s interesting to see how the definition of “tech company” has expanded. Only one or two of these companies create and sell software or technology. These are startups, maybe, and I think we can talk about where the “innovative” companies are. But these are companies that run platforms online, or use the web to sell products in novel ways. They seem very different from most of the companies on the parent comment’s list.

The R ecosystem around cloud services and their APIs still seems immature, so it’s great to see folks working on packages in the space.

I’m not 100% sure if this is providing any new functionality not provided by existing cloudyr projects or is just wrapping them in a new API. I think either is fine, but it would help to better understand why you’d want to use flyio vs, say aws.s3 or the like.

Also, there are some aspects of the API that make me a little itchy. If I’m reading the examples correctly, it seems like flyio_set_datasource sets a global variable and then there are generic functions like list_files that do different things based on that global state?

That seems risky to me, and a more idiomatic approach to this would be to have a function that returns a handle object representing a Google Cloud or AWS service, then have generic functions take that handle and dispatch to appropriate methods.

Even then, namespacing in R isn’t really a thing, and I worry that really plain function names like list_files or export_file are likely to get clobbered by other packages using names like that. For packages like readr that are intended to actually replace large swaths of IO functions, that’s fine. But I’m not sure it makes sense for a more specialized package like this.

Despite that, I do appreciate you all creating and open sourcing this. Like I mentioned, any work on cloud packages is welcome from my perspective! Interested to see how this develops.

Not me. I'd take dplyr and related libraries over pandas any day. I've been using pandas for 6 years and I'm still regularly tripped up by parts of its API.

I understand. I believe I pointed that all out my comment above. I wasn't saying that I find the notebooks superior because they allow for rich & dynamic output, but that I find it superior to RStudio when all you want is a quick exploratory REPL capable of rich/dynamic output. I simply find it easier to fire up a notebook and start noodling around than writing an RMarkdown notebook. That really only holds if I'm not overly concerned with keeping or sharing the notebook. Otherwise, I believe RMarkdown is the better option.

I also tend gravitate towards ESS, and probably split my R development time between emacs and RStudio. I've even written a very kludgy Rmd notebook mode that uses overlays to show evaluation results from code chunks. But RStudio is very well-designed and ESS just doesn't compare feature-wise, sadly.

I mean, as a medium for interactive exploration where you might want graphs and widgets or other rich/dynamic output, I still think the notebook is superior. But as a medium for developing complete, share-able, reproducible data analyses, I do think R has the upper hand.

The Jupyter team deserves every accolade they get and more. The console, notebook, and now JupyterLab are some of the key reasons why Python's data ecosystem thrives.

I think Jupyter notebooks are quite useful as "rich display" shells. I often use them to set up simple interactive demos or tutorials to show folks or keep notes or scratch for myself.

That being said, I do think the "reproducibility" aspect of the notebook is overblown for the reasons other comments cite. Notebooks are hard to version control and diff, and are easy to "corrupt." I often see Jupyter notebooks described as "literate programs," and I really don't think that's an apt description. The notebook is basically the IPython shell exposed to the browser where you can display rich output.

This is where I think the R ecosystem's approach to the problem is better (a bit like org-mode & org-babel). For them, there is a literate program in plain text. Code blocks can be executed interactively and results displayed inline by a "viewer" on the document (like that provided by RStudio), but executing code doesn't change the source code of the program, and diffs/versions are only created by editing the source. At any point, the file can be "compiled" or processed into a static output document like HTML or PDF.

This is essentially literate programming but with an intermediate "interactive" feature facilitated by an external program. RMarkdown source doesn't know its being interacted with or executed, and you can edit it like any other literate program.

Interaction, reproducibility, and publication have fundamental tensions with each other. Jupyter notebooks are trying to do all three in the same software/format, and my sense is that they're starting to strain against those tensions.

Yes, thanks to everyone for keeping it informative and civilized!

I think the constant replies of "Oh there's a way to deal with that." Miss the point. You should keep asking yourself, "Am I fixing a problem that didn't need to be there?" Sometimes, the answer is: No, I do want this structure, and it's worth it overall to write interfaces, etc. to add some polymorphism or dynamism to it where needed. In lots of cases, though, you're just writing stuff to accommodate the language. In lots of languages I feel like I'm fighting an internal battle between static-ness and dynamism. Start with static types or classes, then add interfaces or typeclasses, oh and overload these functions. Now make sure these other things things implement this new interface so they can participate, etc.

Sometimes it feels like a real burden for not much gain over just passing around the basic data (a name, an age) I wanted to deal with to start with. Clojure's proposition is that in many many cases, not getting fancy with the data or over-engineering your problem representation will lead to simpler programs that are easier to maintain, giving you an alternative route to safety and maintenance instead of type-checking.

Like I said, I get why the talk is ruffling feathers, and I think it's fair to feel he's being glib or exaggerating; he is. I've also seen Rich and others discuss these point with more nuance and detail elsewhere so I think I'm willing to interpret them in that more nuanced context.

Re. the Haskell mentions, they all relatively offhand and are usually combined w/ mention of another language, like C++ or Java. (I think this also doesn't help outside understanding since he's trying to talk about disparate static type systems in one fell swoop.)

Re. Rich's feelings about Haskell, there are not only the obvious language influences I cited, but also this very explicit response in the Clojure mailing list (https://groups.google.com/forum/#!msg/clojure/DUKeo7sT4qA/TU...)

"There is no purpose to a message like this.

Everyone, please refrain from language wars. Haskell is a tremendously ambitious and inspiring language, for which I have the highest respect. There is zero point in bashing it (or its users) in this forum.

Thanks, Rich"

Again, I see a lot of what looks like "bashing" to you as a response to an over-fetishization of "correctness," and over-hyping of types. I mean, how many times do you think Rich has heard someone complain that Clojure doesn't have types?

It's also nuts to keep saying things like "he doesn't understand Haskell at all."

Re. dynamism in Haskell, I still don't see this as a compelling point. Those things are unidiomatic and not really comfortable to use in Haskell. I mean, you can put type annotations all over your Clojure code, use typed Clojure, and spec and schema the crap out of your program and get all kinds of safety guarantees. I wouldn't make that an argument that Clojure can solve all your type safety concerns. It's just not a comfortable way to write the language. You can write functional code w/ immutable values in C++ and Java, but nobody does it, because the language makes it hard.

I raise the point about dynlang dismissiveness from Haskellers because I think in large part, the tone of the talk is response to a long history of Clojure being criticized and dismissed for being dynamic. I've used both Haskell and Clojure a lot, been in both communities, and really don't think this talk was the opening salvo of glib dismissals.

I think we both agree the sniping about pattern matching, etc. is not really that interesting, and my sense from the talk is that this is really a very secondary issue. (Indeed I think he makes the point that many features are good if you can get them a la carte, but having the language impose them on all your data can be costly.) It would be great to see Haskellers and Clojurians discussing bigger picture problems that go beyond language features, because the two communities share a lot of values.

It's a keynote talk, not a panel discussion. Most keynotes are expressions of strong opinions.

The fact that there was an entire slide designated to tearing down a series of videos by SPJ felt not only irrelevant, but also disrespectful.

From the transcript:

"Simon Peyton Jones, in an excellent series of talks, listed these advantages of types." ... "And I really disagree just a lot of this. It's not been my experience."

How is that tearing down or disrespectful? I get there were a lot of glib bits in the talk, but as you point out, he's talked about these issues with more nuance at other times. It's a shame that hyper-focus on a couple of thrown off jabs at the costs associated with types is distracting folks from the very useful larger point he's discussion about levels of problems in programming, contexts of programs, and how languages that impose strong opinions about how to aggregate information can be counterproductive.

I think the Haskell community is, overall, very good and welcoming, but smugness does creep in a lot, IME. But if you want to talk about meeting halfway, I find that it's much less common to see static FP folks concede any benefits of dynamic languages (besides that they're "easier" in a kind of condescending way).

I get why the tone of the talk is ruffling feathers. If I were a serious Haskeller, I think I'd be a little miffed too. But I think it misses the forest for the trees, and I think, as I commented before, it's hard to understand the context of his issues without understanding his interest in systems. So I get why a lot of rebuttals have been focused on his somewhat glib representation of certain features, but it's still a little frustrating, because I don't think it's a particularly interesting debate.

I also don't think he referred to Haskell specifically at any point, and really just spoke about algebraic type systems generally. It wasn't in the scope of the talk, and I don't think it'd be a very interesting talk, to compare Clojure and Haskell features. I bet he thinks Haskell is a great language. Clojure takes a lot of inspiration from Haskell: default immutability, core seq functions that look like Data.List; STM, etc. There probably wouldn't be Clojure without Haskell. His whole point is that types, like any other design feature, come with costs. They can be quite heavy and constraining compared to their benefits in certain contexts, and that may not be worth it.

That being said, I don't think it's a compelling rebuttal to say: "If you use Dynamic and fdefertypeerror, Haskell addresses his issues." You'd be run out of town writing Haskell code like that.

Re. parametric polymorphism, he explicitly talks about parametricity, and his take seems to be that he doesn't find parametric types that useful for conveying information or guaranteeing interesting things (to him) about your program. I think he's exaggerating, but I get that it's a response to a lot of breathless advocacy about how informative type signatures are.

Again, regarding his tone in the talk, I get it. But I think this should provide Haskellers a good opportunity to examine how casually dismissive they are of other languages, especially dynamic ones. IME, statically typed FP proponents are much more dismissive of dynamic languages than dynamic language proponents are of types. It's often "Your language is unsound garbage for lazy programmers" vs. "Sometimes the type system becomes an overly-complex constraint on my problem."

As someone who does like types, I'm nonetheless glad that there are folks designing sound dynamic languages and arguing for their usefulness.

There is some basic fundamentals how programming language should work which R just disregards for the sake of being different.

So, you can complain that R feels different from languages you're used to, but it's incorrect to assert that R does things for "the sake of being different." R is simply influenced by a different lineage of languages than, e.g., Python.

Okey indexes start at 1

Languages have had 1-based indexing since Fortran. If your languages is doing numerical computing, you're going to take a lot of cues from Fortran.

Variables are initialized with "a <- 1" okey well for each to their own.

This is how APL does assignment. Like APL, R is an array-based language (all types are vectors/arrays; all functions operate over arrays).

Return requires you to wrap values with "()" so "return x" doesn't work but "return(x)" does. Umm.

R is highly functional. Arithmetic, assignment, and indexing operators are all functions. There are very few "statements" in the language. It's actually highly consistent with that to have return be a function, not a statement. It's actually uncommon to use return in R, since every block automatically returns its last expression.

You should never iterate over a dataframe except when you have to and then there's a plethora of different ways to do it. Some are better, some are worse. But isn't it nice that everyone can invent their own way of doing it? Right?

A data frame implements both list and matrix semantics. This makes sense since, like a list, it is an array of differently-type arrays. But like a matrix it has a notion of rows and columns. Over time, folks have tended to move away from using matrix semantics.

Those are what I can come up with from the top of my head. But overall the feeling I get when I code R is that it's this mystic arcane magic that requires completely new way of thinking compared to other programming languages. And it's very frustrating.

I mean, this is a fair complaint, but it's not a problem with the language. You have to learn it. At its core, R is actually a fairly simple, elegant language that has a few key principles, and some interesting and powerful design features (like delayed argument evaluation).

In "Javascript: The Good Parts", Doug Crockford says this:

"JavaScript is most despised because it isn’t SOME OTHER LANGUAGE. If you are good in SOME OTHER LANGUAGE and you have to program in an environment that only supports JavaScript, then you are forced to use JavaScript, and that is annoying. Most people in that situation don’t even bother to learn JavaScript first, and then they are surprised when JavaScript turns out to have significant differences from the SOME OTHER LANGUAGE they would rather be using, and that those differences matter."

Which reminds me of most people's attitude towards R.

I feel like it does no benefit to the community to simply let it stand just because the person speaking is someone kinda famous.

That's not really what's happening. As I explained in my comment, I've found the technical rebuttals by Haskellers to be lacking.

Yes, of course Haskell can be and has been used to make large, high quality systems. I can say the same for C++. That doesn't mean there aren't costs associated with those languages, and good reasons why someone might want to make different decisions about how to design a language. This is literally what the talk was about: why Clojure was designed the way it was. Not why Haskell is a bad language (it's a great language), but why Clojure was designed differently. Pretending like the type system has no costs and only benefits is not serving the Haskell community well.

I can see why you would feel that, if you hadn't seen Hickey's talk

Not sure what this comment is. I've clearly seen the talk. This pattern of assuming someone who disagrees with you must have less information is off-putting.

What I'm taking away from Haskeller rebuttals is: Haskell has solutions to all your problems, if only you're smarter than Rich Hickey. This is not, to me, a compelling sales pitch.

As someone who really likes Haskell, I've found the response to the talk from Haskellers, like in that thread, really disappointing. The common refrains indicating that Rich doesn't understand Haskell, types, etc. are patronizing and likely incorrect. (I realize he's trying to hit a few targets, from C++ to Java to Haskell in one go, so it's not always clear which he's complaining about.)

The other response I see is that if he were only aware of feature X (mostly row polymorphism), then that solves his issue. Often feature X is some immature Haskell extension, or exists in research or still niche languages. I don't think switching to Purescript is going to solve more than one of his issues, if even that. And the last thing I've seen is a bunch of folks trying to torture the crap out of Map to prove him wrong (not really) about some offhand point or another.

By and large I've seen a lot of (pedantic) sniping at specific phrasings without much attempt to grapple with the larger points.

The casual dismissiveness of users of dynamic languages (programmers use them because "dynamic types are easy" says one commenter -- about a talk from a guy who has famously thoroughly dissected the notion of "easy.") along with the inability to actually engage with the broader ideas has kinda turned me off the Haskell community.

Lastly, it seemed clear to me from the video and transcript that he was saying performance optimization, not (just) Intellisense, is a clear win for static over dynamic types. And "Intellisense" is kinda just a shorthand for static analysis generally. Again, the point is not that types don't have benefits, its that those benefits come with costs. I think Haskellers often underplay the costs associated w/ dealing with the type system, underestimate how little reach the type system has in an open system where data's flying around arbitrary services, and overstate how much preventing internal inconsistency bugs solves all problems.

I like Haskell, OCaml, F#, etc. I think they make a lot of hard things simpler and help me reason about certain programs better. But they're not panaceas.

Generics can make serialization easier in Haskell, but that's not exactly the point. The point is, once your Haskell program is done with that data, it's getting tossed into a message queue or database, or whatever, that doesn't really care or have any concept of what typeclasses it implements, whether one of its constructors is an Either, etc. In open systems you don't really get to decide who consumes your data or how--your program can't communicate anything other than data to them--and so you often don't have a way of enforcing your types on eventual consumers. Haskell has strong opinions about how it thinks data should be represented and aggregated. But in large open systems, as the saying goes, "opinions are like aholes; everybody has one."

When I think about the popular tools for moving data around large open system: the message queues, key-val stores, pub-subs, etc. --- it seems to me that the idea moving and communicating about types and objects over wires has largely been a dud. Thinking RMI, OODBs, etc. It's just hard to get other people (tools, services) to care about how you've decided to organize the entities in your program. It's a lot of work, and the benefits over throwing mostly "plain" data may not be compelling enough.

Again, I keep coming back to his term "parochialism" and why he's focused on it. I think it's an under-appreciated point amongst all the language wars.

Agreed that's bad. But then if the transit/messaging/persistence components of your systems are independent of the type system (good), to really use your type system you have to do work pushing things in and out of it, in return for type safety (and sometimes not much) that only lasts until the border of your program. It's really easy to over-engineer your types because you want really want to pin down the representation of your problem in the idioms the language gives you. ORM (ab)use is a good example of this, I think.

I've often made the mistake myself of architecting a too-clever type or class system for my problem, and then been faced with writing tons of crap to wrestle it in and out ofprotobufs, etc. that needed to be more general than my problem. When my program was running, it was like, woo, I made some illegal states unrepresentable, which felt great! But I could almost never do that in a way that didn't quickly reveal itself as too brittle.

I like types (mostly). I wish gradual/partial typing was a better solved problem. Clojure's goal is to make it so that you don't over-engineer and tangle up your systems by passing around simple immutable data. If you keep your system nicely decoupled, the types, which are good at finding when I've forgotten a coupling in my code, seem less valuable to me.

I think type systems that try to "close" aggregates (i.e. saying "an Employee is these fields that have these types and no more") kind of do contribute to the problem. I sort of agree that static types are not "causing" such problems. They don't "cause" bad design, but they tend to make it too easy to set bad designs in stone (and most designs are bad in some way). It's not so much about types causing problems or being bad, but having costs, and thinking hard about those costs vs. the benefits. Different folks will, and should, make conclusions for their problems.

I read him as exaggerating his critique a bit because types are often oversold (static type people can be really dismissive of dynamic languages). But I think he's mostly making a "no silver bullet" kind of argument.

I love this talk, but it does throw out a lot of complicated ideas somewhat loosely, so I get why reactions and interpretations are all over the place.

I think a good companion to understanding this better is Rich's talk on "The Language of the System" (https://www.youtube.com/watch?v=ROor6_NGIWU&t=2810s).

I interpret his overarching thesis as this:

Most "situated" programs are in the business of processing information according to complex and changing rules, in cooperation with other programs (i.e., a system). Many languages, though, are overly-concerned with how they represent data internally: classes, algebraic types, etc. This "parochialism", he calls it and "concretion" about how data aggregates should work, make them hard to adapt when the rules, data, or other parts of the system change, and make it hard for their programs to work in systems. At some point your Java class or Haskell ADT has to communicate its data to other programs that have no notion of these things. So you end up w/ a ton of surface area in your code with mappers and marshallers totally unrelated to the content of the data and purpose of the program.

The idea behind Clojure is to provide easy access to simple, safe, and relatively universal structures for holding data, and a library of functions to manipulate those structures. Its "big picture" design bits are about providing semantics for multiple "programs" (from threads to services) in a system to operate on data robustly and reasonably (concurrency semantics, time and identity models, pervasive unadorned data, etc.) At some point you're going to be sending this program's data over a wire to another program, and things like "a map of strings and numbers" is pretty straightforward to transport, while a sum type implementing functor with a record data constructor that contains a Maybe SSN is not. It overly couples the underlying data to the language's representation.

The plus side of doing this is that the language can check internal consistency for you. The downside is that you're carrying a lot of baggage that you can't take with you over the wire anyway. Communication in systems is also why Rich thinks making "names" for data first class is important. Existing strongly typed languages can sort of accommodate this, but don't really privilege names.

So I think a lot of strong typing advocates are upset because they think Rich is saying types don't have value within programs. I don't think that's right. I think he's saying they have very limited value in open systems, which makes their costs often overwhelm their benefits in the individual programs within those systems.

In general, I feel like the debate has been about examining Rich's claims in the context of programs (is Maybe String good or bad, etc.), whereas he's really interested in what works in systems. I think that's indicated by his focus on the term "parochialism" which I have not seen a lot of folks address.

It's a sort of lisp-y idea. Arguments passed to functions get "quoted", so the function can get or change data about the expression and its scope/environment before evaluating it. Like others have mentioned, it's what make R quite good for developing DSLs like R's formula language or dplyr. (And other conveniences, like auto-labeled plots, etc.) But similar to lisp macros, it can make for unpleasant surprises if not used wisely.

If you look at attempts to do this stuff in python---e.g. patsy, which emulates R's formula DSL, and there's another project that emulates dplyr I don't recall the name of---you see they have to resort to parsing and eval'ing strings instead of working on expressions (language objects that represent ASTs), which is not nearly as nice or safe.

Edit: But just to emphasize your surprise -- yes, you can definitely be surprised by delayed evaluation in many contexts if you're used to more traditional languages.

    > y <- 10
    > wat <- function(x=10*y) { y = -y; x }
    > wat()
   [1] -1000

    > y <- 10
    > less_wat <- function(x=10*y) { force(x); y = -y; x }
    > less_wat()
   [1] 1000

I've been using Triplicate (http://practicaltypography.com/triplicate.html) for a few years and it's still my favorite by far. Not free, but really reasonably priced. (Note there's a special variant suitable for coding, you can see examples in Racket in the PDF specimen.)

I picked up _The D Programming Language_ on a whim---having not written a line of the language---and found myself reading it straight through in couple of evenings. It's a really clear, insightful, and funny book about the language's design. Since then I've been doing all my hobby projects in D. The language is really fun to code in, mixing really powerful constructs with nice syntax/good ergonomics.

Nice to see this, a relatively complete intro/tutorial to the language for newcomers to explore.

This is an interesting project, and it's illuminating to see what it takes to emulate some R features in Python (custom infix ops, non-standard evaluation, dataframes as namespaces/envrionments, etc.)

But I feel like it would be better to use method chaining for the piping of transformation rather than overloading dunder method operators. It would preserve one of the nice things about dplyr -- composing complicated transformations from a simple vocabulary, but more pythonic. This is a relative weakness I see in the design of pandas and would love to see ported over.

But also, dplyr is a thing that really goes beyond pandas. It's really an elegant, SQL-like DSL for transforming (mostly) arbitrary data. In this way it's more like LINQ than a specific implementation/API of a data structure.