eshvk

You are right, my mistake.

H1B:

The problem now is that the visas seem to be harder to obtain and have a longer delay. I know people who were stuck waiting for multiple months to do a transfer.

L1:

My understanding is that this doesn't provide a path to a green card.

One thing I have observed is that engineering roles at national labs tend to have a lot of paperwork needed for an interview. I can understand a faculty role requiring recommendation letters, a long application form to fill up but an engineering role requiring all that makes it rather annoying.

Not sure if Oak Ridge does this tho. :)

In the broader sense, I guess my question is what are the next steps to build on this basic recommendation system? What is a good way to serve recommendations based on user behavior in near-real time and how do these systems take feedback to improve the models.

In the past, I have helped build Lambda architectures where we use a batch model to build a content vector space, build estimates of users in batch, update those in realtime (using PubSub/Kafka) based on user feedback.

Other online mechanisms could be to use Contextual Bandits: e.g. use context in terms of user interactions with the several arms of the bandits being recommendation choices etc. This interaction data can be used to continuously improve your policy. Of course, the key benefit over a Matrix Factorization setup where the interaction matrix is continuously rebuilt over time based on new data, is the in built exploration which minimizes regret.

Not really sure what your central question then is: The general theme for building a recommender system architecture is:

1. Decide whether you are okay with a batch approach or an online learning approach or a hybrid.

2. Start simple with a batch approach (similar to what you are doing):

a) Get features ready from your dataset (assuming you have interaction data) : Pre-processing via some big data framework (Map Reduce, Data flow etc)

b) Build a vector space and nearest neighbors datastructures.

c) Stick both into a database optimized for reads

d) Stick a service in front of it and serve.

Once you are happy with 2, you can try out variations involving either online updates to your recommender system which involves changes to the type of database you might want to optimize. etc

There are roles called Machine Learning Engineers/Research engineers. The work does require enough background in ML; as in it is non trivial to implement an algorithm, consider approximations while scaling without knowing how the math works.

Although, you might be able to get away with not knowing recent ML work by having a solid education in linear algebra, probability and statistics.

I work in this field. I can tell you that the maths involved is an easier (and more exciting) learning curve. A lot of software engineering already involves the tools necessary for getting good af math: example SVG transformations and matrix algebra, learning how to manipulate abstraction in operating systems etc.

I'd suggest an approach where you use your (presumably good) software engineering skills to join an engineering gig in a data science team. Then, gradually pick up tools, slowly pivoting into fully fledged ML/AI. It is definitely possible as a career path.

Spotify extensively uses Scala for data infrastructure and ML feature engineering.

I work in Sweden. I don't think I have seen ever anyone fired. What I have seen is some of the best hiring processes. The process looks at the candidate values, and skills. And there is a discussion about what we expect from her, and what she expects from the job.

I have worked for a few years in a company that is based in Sweden but also has a very large U.S. presence. Couple of points:

1. The recruiting process is involved in Sweden is more involved than in the U.S, agreed. However it is not perfect. False positives do enter the system. Folks there have started getting junior candidates intern/contract as a way of evaluating them. This unfortunately has its own bias.

2. Yes, I think it is impossible to fire people in Sweden. What is typically done in those rare situations is to shuffle you in such a way that you don't do any work. Imagine a head of a team suddenly head in title but with no reports.

3. On the U.S. side, interviews are way easier but then at will employment takes care of that in those rare situations.

There is/was a option available for F1 Visa students called OPT (optional practical training). It could in theory run up ~24 months depending on the company having everify or not.

Probably will start with either linear algebra or maybe set theory?

This can work. I spent a year or so before grad school at work, I brushed up by spending a summer drilling through a Linear Algebra book.

A new thing I have been trying out recently is find an application, find something I am confused by, dig deeper through the layers of mathematics till I uncover the basics of the confusion, then re-learn and work up.

So BFS vs DFS. The latter seems more like up my alley. YMMV

I think the poster assumed that data science/computational work could provide the mathematical background the OP is looking for.

Because Anaconda is a package of tools that include numpy,scipy (python numerical and scientific computing) and a notebook (a "live editing" utility).

While this might help, I wonder if this is too roundabout a way to learn math.

Sure I have a new found love for math, but I'm not going to be accepted as a mathematician ever without the rigor of a formal education

I think there are two separate things here:

1. Being a research mathematician requires a degree of expertise which is easier to come by with formal education.

2. Knowing how to use APIs and make correct distributional assumptions; despite the bullshit fed by our industry, it is not easy or non trivial to design a completely idiot proof API. So having know-how of how the math works under the hood is helpful even if you are going to just use the API.

Also things like qualitative user testing, a/b testing would go into every single pixel of his design. I am not particularly sure if the most important separation of the music listening population is into hoarders vs nomads. You'd evaluate that, make a decision. Things might get muddled there.

My theory is that they have a few people manually curate a list with good songs for a while to train an AI, but when they finally turn control over to the AI it falls on its face and starts adding shitty cover band music to the list.

A curated playlist (not discover weekly or release radar for example) is editorially curated and driven by a human. Doesn't involve an AI for now.

Noise cancelling headphones don't really drown out conversations. You should get in-ear headphones.

Many of the people share similar backgrounds

Not sure where you get that from. I find significant diversity in academic backgrounds amongst the people who do ML/AI. Some folks in my company include: Mathematicians, Biochemists, Physicists and Computer Scientists. I used to work with an awesome person who had a Poly Sci background. So yeah, things are all over the place.

Apparently the new hotness is AI. The last five years have seen titles with similar(ish) roles evolve from Data Scientist -> ML Engineer -> AI Engineer

mathematical/academic concepts like big O

I hope I am not nitpicking here but ain't nothing wrong with knowing the time/space complexity for code you'd write. Especially if you are a "senior software engineer".

is just not built for Deep Learning

What does a "built for Deep Learning" GPU do that the K80 doesn't do?

Hey let's be honest, most academics (that I know) still don't even use LaTeX (or refuse to do so).

What field? TeX is pretty much de rigueur in Math/CS/Physics graduate schools in the U.S.

I work at Spotify on ML. I spent quite sometime working on our production vector model pipelines.

We have three components:

1. Training data: Scalding pipelines (could be any large scale production framework) to preprocess training data and reduce it to a few hundred gigs.

2. Run latent vector models (Matrix factorization, Embedding models etc) to generate item vectors. This stage is designed to be non specific to framework. We use typed avro objects to communicate with the previous stage.

3. Production deployment of models. Essentially glue code that ensures that models are versioned, atomic (user, music vectors are all from the same batch) and deployed into music recommendation and realtime frameworks.

Read here[1] more.

[1] https://labs.spotify.com/2016/08/07/commodity-music-ml-servi...

Well no, there doesn't have to be that many people in the interview, especially the first one. I've never heard of recruiters being there for the interview at all.

The recruiter's job is to make sure the candidate gets to the right office room in time; makes sure that if an engineer drops out that there is someone else to pick up the slack. Also, if the candidate has a public freakout or decides to get nosy and peek at someone's computers, it is their job to be nice and navigate the candidate to a more appropriate scenario.

Depends, some might allow the interviewers to start late

My point is this doesn't scale beyond a certain point. I have complete sympathy for job seekers but I think you are underestimating the amount of grievance that can builds up when one is forced to interview you. As a potential interviewee, I'd bite the bullet, do enough preliminary talks/coffees to figure out if this is what I really want and then go interview at a time that is mutually convenient.

go after normal business hours

Think about it from the other perspective, you want to do an interview after normal business hours. This means that a recruiter, several engineers and a culture interviewer have to also be present after work hours. Now scale this up to a few dozen candidates; I wouldn't want to work in a place that does this on the regular.

I run models on large scale user data (for recommendations). A couple of hundred gigs is what "cleaned" data looks like. This might need to be joined with metadata. This cleaning, joining is easier to do on Map Reduce than wait a few days on a beefy computational machine.

Most Deep Learning jobs, and even relatively mundane Data Scientist jobs nowadays, want a PhD from my experience.

Uhh..No. I have been doing this for 5 years or so. I don't have a PhD; most people we have hired don't have PhD. Some who write NIPS papers (very different from a "Medium thought piece") on their spare time. Now what we optimize for is relevant experience and the ability to not just throw a framework at something. That is highly correlated with having worked on this for a while or have strong math skills. Guess, what? Some of those who have these skills have a PhD. Some, not all.

Given that AI is highly driven by research right now (no "if" about it), what talent would want to work somewhere where they can't publish research papers?

Ehh. I would question that doing research is necessarily highly correlated with wanting to publish papers all the time. A lot of the work during product development is a mix of AI, UX and engineering. The entire output might definitely fall in research but the individual components of it might require way more fleshing out to bring each to publishable quality. Considering that we don't particularly have the publish or perish doctrine of academia, we are not particularly forced to publish at the same rate as there.

India has a huge movie industry (which is unusual enough) but it gets almost no attention from outside India except by expats.

Do you have enough evidence to support this claim? Anecdotally, I have seen a lot of folks in Africa, the Middle East obsess about Bollywood movies. To me, anime (outside Asia) is a niche whose obsession is heavily correlated with young American/Western/white men. Could possibly be a reason why tech (with its demographics) has a lot of anime lovers.

You need an H1B. You would need a job and sponsorship from an employer. So, yes apply for a job and discuss/organize visas.

H1B where their employer is their board? But it's still a lottery

I don't understand how you plan to start a business when you are being employed by someone else? As in I don't think you can be listed as a founder in the company that is offering you a H1B

O-1? But it's supposedly a very hard visa to get?

Yes

EB-5? But you need $500K-1M

Yes.

Ultimately, it is not easy and only a few can work (not start a business) in the U.S. YMMV and you should probably talk to an Immigration Lawyer.