gjreda

The recent changes have gotten absurd

Grab some baked goods at Hewn (in Evanston), visit the Bahai temple, and then walk across the street to Gillson Park to wander the Lake Michigan shore and eat your goods from Hewn.

The easternmost portion of Northwestern's campus also has a nice walking/biking path along the lakeshore with a great view looking back towards the Chicago skyline.

https://gregreda.com/

I've mostly written technical, code-centric posts on Python, ML, and data science. Some of my early posts (2013) were wildly popular at the time and hit the top of HN and various subreddits.

I haven't written much recently, but I've been trying to branch outside of technical posts as I felt like my profession had started to become too much of my identity.

The post I'm most proud of:

- https://gregreda.com/2022/11/30/this-ones-for-me/ - Feeling pride and catharsis after years of bad health luck (leukemia, bad bike crash, cardiac arrest).

My most popular posts:

- https://gregreda.com/2013/03/03/web-scraping-101-with-python... - Web scraping tutorial using Python and beautifulsoup

- http://www.gregreda.com/2015/02/15/web-scraping-finding-the-... - Another web scraping tutorial with Python, but this time for sites that dynamically load content

- https://gregreda.com/2013/10/26/intro-to-pandas-data-structu... - The start of a series of posts on Python's pandas library

- https://gregreda.com/2013/07/15/unix-commands-for-data-scien... - Some useful unix commands for data processing

- https://gregreda.com/2015/08/23/cohort-analysis-with-python/ - Tutorial on doing cohort analysis using Python and pandas

- https://gregreda.com/2017/01/07/freelance-data-science-exper... - My experience as a freelance data scientist

- https://gregreda.com/2018/02/04/hiring-data-scientists/ - My approach to hiring data scientists (though my thoughts on this have evolved over the last five years).

I initially built this same "chat with PDFs" prototype with LangChain and qdrant. I then rebuilt it from scratch for the sake of learning and comparison.

Some context: I've been a jack-of-all-trades data scientist / machine learning engineer for the past 15 years (officially titled as an MLE the last four years).

I share that only because I think it plays a role in how I'm typically accustomed to working.

1. I found LangChain to be overkill for this use-case. While it might allow some to move more quickly when building, I found it to be cumbersome. My suspicion is this is largely because of my background - I understand how to build much of what's "under the hood" in LangChain. Because of this, I think it felt overly abstracted and I found the docs difficult to navigate and sometimes incomplete.

2. I used Qdrant via their docker image and it was simple to setup and start using. I didn't try to push the limits with it, so I can't say anything about performance. Because Qdrant runs as an http service, I found that it didn't fit well into my workflow - I'm accustomed to being able to visually inspect my data inside the interpreter, debugging, trying out commands, interacting and experimenting with my results, etc. Again, my suspicion is this is my own bias in how I typically work. Qdrant otherwise seemed very nice.

3. LanceDB felt powerful yet lightweight, and fit well into my workflow. It was far more intuitive for me. It was as if sqlite, the python data ecosystem, and a vector database had a child and named it LanceDB. Under the hood, it's built on Apache Arrow and integrates nicely with pandas, allowing me to seamlessly go from LanceDB table on disk, to pandas dataframe, and into some analysis or investigation of my LanceDB query results. This line [1] is a great example of why I liked it. This feels nicer to me than the world of API params and HTTP requests.

1. https://github.com/gjreda/scratch-pdf-bot/blob/main/gpt_pdf_...

Not specific to this model, but beyond the large players (OpenAI, Cohere, etc) are there any free hosted versions of the open(ish) LLMs? Even the smaller 7B parameter ones? I'm prototyping out a project and using OpenAI for now, but it feels like there has to be a hosted alternative somewhere.

I spent some time today exploring HuggingFace's Inference API but if the model is sufficiently large (> 10gb), HF requires you to use their commercial offerings.

I recently prototyped out a "chat over PDF documents" project.[1] I opted to use LanceDB for vector (embeddings) storage and retrieval and found it really nice to use.

I'm working on using it in a large project now.

[1] https://github.com/gjreda/scratch-pdf-bot

5/1000 colonoscopy patients have complications (some fatal) which is way higher than the base rate for colon cancer.

Can you provide a source for this?

At least for Twitter, I think this still happens if you move everyone into a list and only view the list.

Expectations are overwhelmingly for 75bps. Prior to yesterday, 50bps was expected.

- https://www.cmegroup.com/trading/interest-rates/countdown-to...

Imatinib (Gleevac) revolutionized treatment for patients with chronic myeloid leukemia (CML). Prior to the drug’s discovery, CML patients generally had seven years to live (possibly less depending on how advanced the cancer was). Now their lifespan mirrors the general population.

I’d highly recommend the book The Philadelphia Chromosome if you’re interested in learning more.

Recently wrote some code to scrape a friend's reviews and ratings from Goodreads. Maybe it'll be useful to folks here: https://gregreda.com/2020/11/17/scraping-pages-behind-login-...

Here's an excerpt from their October 2019 letter to shareholders. TL;DR - they're a public company and the markets told them they needed to keep growing in order to survive.

For restaurant inventory, we will rapidly expand our recent pilots of putting non-partnered restaurants on the platform. For reasons we’ve discussed many times, we believe non-partnered options are the wrong long-term answer for diners, restaurants and shareholders. It is expensive for everyone, a suboptimal diner experience and rife with operational challenges. With that said, it is extremely efficient and cheap to add non-partnered inventory to our platform and it can at least ensure that all of our current and potential new diners have the option to order from any of their favorite restaurants now, even if it’s not the best solution. By leveraging non-partnered options, we believe we can more than double the number of restaurants on our platform by the end of 2020.

At the same time, because we know that partnered relationships are critical to the long-term success of this business, we will be investing aggressively in our independent restaurant sales organization to support converting as many of these non-partnered restaurants to partnered relationships as quickly as possible and to take advantage of other innovations in the restaurant space, like virtual restaurants.

https://s2.q4cdn.com/772508021/files/doc_financials/2019/q3/...

This was GrubHub's original model. Restaurant's don't want to pay a flat fee when they don't see enough orders they feel is necessary to justify said fee.

[1] https://youtu.be/a2oTps7tKS4?t=893

Having recently road tripped across the country for a move to SF, I found myself admiring the design of national park signage. It feels very timeless and iconic.

https://i.pinimg.com/originals/13/76/e2/1376e2e5690719701689...

Does this change when the two options are perfect substitutes?

As a rider, there's no "cost" to having both apps, so I'll just check both.

Similarly, as a driver, I'll just sit with both apps open and see which one I get a rider on.

I guess I don't really have a reason to post any of this, other than it helps to get it out there

I know that feeling well. Best of luck to you.

I agree. Hadley Wickham (a very prolific author of important R libraries) wrote a great paper about this method using one of his libraries. I'm a Python + pandas user, but his paper really helped me understand the approach better: https://vita.had.co.nz/papers/plyr.pdf

Sprout Social | http://www.sproutsocial.com | Engineering & Non-Engineering | Chicago, IL & Seattle, WA | ONSITE | Full-time

Sprout Social builds social media management tools for businesses of all sizes. We are built on the idea that the world is better when businesses and customers communicate freely. We exist to help streamline and enhance those conversations — with customers, prospects and enthusiasts.

Some openings:

    * Front-End Engineer (Seattle, WA)
    * Senior Front-End Engineer (Seattle, WA)
    * Senior Software Engineer - Platform (Chicago, IL)
    * Staff Software Engineer - Platform (Chicago, IL)

Our platform team uses Java, Python, MySQL, and NSQ, while our front-end team uses React, Redux, Ember, ImmutableJS, and Gulp, all to build highly scalable software that is used by more than 20,000 organizations around the world. Companies like Dropbox, Zendesk, Fender, Brooks Running, Seattle Cancer Care Alliance, and Evernote rely on our products to create stronger relationships with their customers.

If you're a creative, highly motivated, and inquisitive learner, we'd love for you to come build great software with us.

https://sproutsocial.com/careers/

Clickbait post title leaves off that this is an estimate from a research firm. Title makes it seem like a sure thing.

In case you're looking for more, this tutorial series hit the front page of HN a few years ago: http://www.gregreda.com/2013/10/26/intro-to-pandas-data-stru...

Modern pandas is a bit more idiomatic now though: https://tomaugspurger.github.io/modern-1.html

Sprout Social | Senior Data Scientist | Chicago, IL | ONSITE | https://sproutsocial.com

Our team uses Python and its data stack (pandas, numpy, scikit-learn) along with a little Java to drive decisions and power software that is used by more than 17,000 brands around the world. Companies like Microsoft, Zipcar, Hyatt, Google, and Zendesk rely on Sprout to create stronger relationships with their customers through social media.

We’re looking for curious, analytical, and creative people to help utilize the vast amount of data we have. If you love finding ways of using data to build better products and solve problems, we’d love to talk with you.

Responsibilities:

    - Build predictive models in support of product and business goals
    - Conduct research to surface new opportunities and understanding
    - Build tooling for all of the above
    - Mentoring, teaching, and helping to level up the team

    - You have professional experience working on data-driven problems. Examples might be classification, recommendations, churn prediction, topic modeling, fraud detection, etc.
    - You’ve built production machine learning pipelines in Python or Java.
    - You have a strong understanding of statistics (probability, experimental design, bayesian methods).
    - You enjoy thinking about business and product, and how data can be used to inform or improve both.
    - You understand, appreciate, and can explain the tradeoffs between simple methods and complex models.
    - You probably studied a technical, quantitative, or analytical field throughout school, but also might have taken a different route. We’re looking for at least a Bachelor’s degree or equivalent.

    - Experience working with text data and using natural language processing methods
    - Experience with Apache Airflow, Spark, or AWS Data Products (EMR, Redshift, Kinesis)
    - Experience with deep learning toolkits, such as Keras, Theano, or TensorFlow

Sprout Social | Sr. Data Analyst | Chicago, IL | http://sproutsocial.com/

Sprout Social builds social media software that is used by more than 17,000 brands around the world. Companies like Adobe, Pivotal Labs, Google, Marvel, and Evernote rely on Sprout to create stronger relationships with their customers through social media.

    About the role
    --------------
    - The role is cross-functional and will work extensively with our Data Science, Marketing, and Optimization teams.
    - Our Data Science team uses Python and its data stack (pandas, numpy, scikit-learn, airflow, etc.), but we like R users too.

    Responsibilities
    ----------------
    - Use rigorous analysis, statistics, and a good deal of consideration to help us think about our funnel.
    - Plan, analyze, and report out on split testing experiments.
    - Use data to inform and influence product roadmaps and departmental goals.
    - Build tools to help us make better decisions and automate analyses.
    - Conduct research to surface new opportunities and understanding.

This is a welcome addition. I've run into bugs in projects before and wanted to "+1" a thread, but it always felt like spamming the maintainers.

It'd be cool if they added a way to search through your list of reactions. This would allow you to effectively comment on an issue in an OS project, while simultaneously bookmarking it, so that you can go back and commit a fix when you have a free moment.

For anyone interested in data analysis in R or Python (specifically Wickham's dplyr or McKinney's pandas), I can't recommend reading Wickham's paper on the split-apply-combine methodology enough. It really helped me understand working without looping.

The paper's examples might be slightly dated as they use dplyr's predecessor, plyr. I imagine they're pretty similar though.

http://vita.had.co.nz/papers/plyr.html

Are drivers really under Uber's control when they can easily have both Uber, Lyft, and any similar app open at the same time?

I think in major cities, both Uber and Lyft are becoming commodities - both of them have ample enough supply that I just choose the cheaper option.

Conspiracy theory: Amazon's sorting is intentionally bad so that you have to look through other items you might purchase, much like grocery stores organize so that you have to work your way through them.

I don't actually believe that, but I do find myself thinking of other things I've needed/wanted when browsing Amazon.

There are a few services that work with Twitter to do this - Pocket comes to mind. I know their Chrome extension will place "Pocket" buttons on each tweet allowing you to save them for later. Similarly, a long button press on iOS will allow you to send a link from within Twitter to Pocket.

This is a shame. IMO, Zach Lowe's the best basketball writer out there.

I see it as teaching fairly basic business and personal finance concepts. In a league where 60% of players go bankrupt within five years after leaving the sport [1], this is likely pretty enlightening for some readers.

[1] https://en.wikipedia.org/wiki/Personal_finances_of_professio...

On the day Moments was released, I followed the "Moment" of the Astros-Royals Wild Card (baseball) game. I loved it - it pulled relevant tweets into my timeline and let me follow the game with people doing the same thing around the world. When the game was over, those tweets (and users) were gone from my feed. It was perfect.

Then, the next night, I wanted to do the same thing for the Cubs-Pirates Wild Card (baseball) game, only it was never a "live" moment on Twitter.

Right now, the Blue Jays are playing the Rangers in a "win or go home" situation, but again it's not even a live event for me to follow.

I love Moments from a product perspective, but I can't help but think this was rushed out too soon - right now, there are only two "live" moments. And because it's been rushed out, I can't help but think it's going to cause the same problem Twitter has been trying to fix - that users are going to try it, not get it, and never come back.

Moments should be about _live_ content. It seems focused on things that were a moment earlier in the day or week.