kmax12

We stream new data in as the sources publish it. typically, every 5 minutes.

We collect from a lot of sources, so nearly every minute something is updating.

if this type of data is interesting to you, here’s a site I’ve built that tracks like grid data across the US and Canada: https://www.gridstatus.io/live

We have generation mix, load, and pricing data. Both real time and historical

This article discusses the “deregulated” energy markets that allow many different participants to be involved in the generation of electricity.

One consequence of having many market participants is the availability of data that is published to make the markets function.

If you’re interested in seeing more about the real-time operations, I built a site that tracks all this data: https://www.gridstatus.io/live

thanks for heads up — fixed now

Grid Status | Senior Software Engineer | Remote or Hybrid Chicago | Full-time | https://www.gridstatus.io/ | $150k+ salary & equity

Grid Status is one of the most widely used energy data platforms in the world. Our data has been used in reporting and analysis by the Wall Street Journal, New York Times, the Federal Reserve, and by thousands of users who regularly visit our products every month.

As one of the first members of our engineering team, you will have a hand in both defining and implementing the product we deliver to customers. No experience in the energy industry is necessary—just a strong eagerness to learn. Our team’s passion and expertise will help you quickly get up to speed.

Our stack is mostly: python, react, postgres, snowflake, AWS

Job posting: https://gridstatus.notion.site/Senior-Software-Engineer-7a2a...

To apply: email us at jobs@gridstatus.io with a resume, link to any projects, and why you're a fit for the role.

You may need to check the time zone as some pages automatically adjust to local time of your browser. It happened at 7am CT.

I helped write this! The data behind all the graphs in this post is here: https://www.gridstatus.io/dashboards/ercot-nuke-trip-july-20...

If you like grid data, we have a lot more info for the entire country here: https://www.gridstatus.io/live

Grid Status | Senior Software Engineer | Remote or Hybrid Chicago | Full-time | https://www.gridstatus.io/

Grid Status is one of the most widely used energy data platforms in the world. Our data has been used in reporting and analysis by the Wall Street Journal, New York Times, the Federal Reserve, and by thousands of users who regularly visit our products every month.

As one of the first members of our engineering team, you will have a hand in both defining and implementing the product we deliver to customers. No experience in the energy industry is necessary—just a strong eagerness to learn. Our team’s passion and expertise will help you quickly get up to speed.

Our stack is mostly: python, react, postgres, snowflake, AWS

Job posting: https://gridstatus.notion.site/Senior-Software-Engineer-7a2a...

To apply: email us at jobs@gridstatus.io with a resume, link to any projects, and why you're a fit for the role.

It’s a wide range of sources but basically the different Independent System Operators in the US and the Energy Information Agency (EIA) All of our data scrappers are open source: https://github.com/kmax12/gridstatus

The US electrical grid is changing fast. I built a site to track instantaneous solar generation and other records in real time across different parts of the country.

New solar records are done until next summer but still interesting stuff happening. For example, California hit a new battery charging record a few weeks ago.

All the records and more real time information about US grid are here: https://www.gridstatus.io/records

I am one of the people who built this page and the Grid Status site. Cool to see this posted!

Ultimately, the solar eclipse this weekend came and went without any noticeable impact for consumers of electricity. In my opinion, this is a very notable feat.

A well-planned and executed response by grid operators to a predictable eclipse bodes well for a future where events like a particularly cloudy day, blizzard conditions across a large area, or widespread hail damage to solar farms could reduce generation by similar MW values in the middle of the day.

As the grid becomes a more dynamic and volatile place, making operational and investment decisions based on data is more important than ever before.

If this sort of stuff interests you be sure to check out our other dashboards: https://www.gridstatus.io/home

fixed!

Indeed! One of the reason I made this site is to make it easier to track all the prices in one spot

It depends on the source of the data. For the regions with the pie charts, those are Independent System Operators (ISOs) or Regional Transmission Organizations (RTOs). They publish much more data and is more real time than the other regions, so we have different pages to display them.

I am too!

We focus on the United States, so we can have the deepest coverage for each of the regions. In some cases that means were the same as Electricity Maps. In other case, you'll see our data is more real time without relying on estimations.

We have a lot more than generation mix data. For the wholesale energy markets, we also have all the pricing data[1]!

In case you're curious we also have an open source library: https://github.com/kmax12/gridstatus

We'd love to expand, but every new region is hard to support in the same depth as what we currently have in the US

[1] https://www.gridstatus.io/datasets?filter=lmp

hmm, ill have to look into that. in that case, check out this listing of all the areas we support: https://www.gridstatus.io/eia.

also, if anyone else has this problem please comment / email me max@gridstatus.io

If you're curious about this data for the United States, I built a site that lets you track renewable production, as well as many other interesting datasets about the US electrical grid.

https://www.gridstatus.io/map

we report the instantaneous load records while that page is reporting the hourly average. I wouldn’t say the results are that different. They point to about the same times even if numbers have a small delta.

My focus right now is to make this data more available and easier to use. Haven’t done much to monetize yet. If you just want that data source, happy to get it to you for free. Shoot me an email at max@gridstatus.io or sign up for a free api key and grab it from the site

At peak, renewables provide up to ~90% of California's electricity

In fact, renewable generation regularly hit more than 100% of load in California during April and June of this year. The peak was 132% of load [0]!

How can generation be more than 100% of load? California was exporting power to other regions.

We track all this data and more across the United States at Grid Status: https://www.gridstatus.io/home

[0] https://www.gridstatus.io/records/caiso?record=Maximum%20Ren...

I’m working on one for the United States! It also has a lot more data about grid operations like wholesale pricing of electricity

https://www.gridstatus.io/

Long-term trends clearly demonstrate the energy grid’s transition to renewable energy sources.

However, renewables like solar and wind come with unique challenges due to their intermittent nature. They are more variable, harder to forecast, have location constraints, and can benefit from battery storage. These factors lead to a more dynamic grid than before.

For instance, several regions in the country provide five-minute updates on their energy generation mix, enabling near real-time observations of renewable energy effects throughout the day

For example, numerous regions across the country provide updates on their energy generation mix at five-minute intervals, allowing for close to real time observations of these effect of renewables throughout the day.

To help those involved in the energy transition, I created an open-source project called Grid Status (https://github.com/kmax12/gridstatus) that provides fuel mix, wholesale pricing, load, load forecasts, and more.

Additionally, I've developed real-time visualizations to make this data more accessible and easier to comprehend: https://www.gridstatus.io

I hope making this data more accessible and understandable will accelerate the transition away from fossil fuels.

Interesting to see heat pumps come up more and more these days. As a lot of comments here point out, heat pump technology is already pretty good, even in cold climates.

It seems to me a lot of the barriers to adoption in the US are lack of awareness from consumers and widespread support from installers. The equipment for a heat pump shouldn’t be much more expensive then an air conditioner since they share so many parts, but that isn’t the case in practice.

I think government regulations that encourage heat pumps manufacturing and installation are part of the solution. For examples, Biden administration recently issued orders to use the Defense Production Act to produce heat pumps [0] or NYC banning new natural gas hook ups for heating [1].

The other part of heat pumps adoption is making them exciting for consumers. It feels like if you get the right combo of all of that, heat pumps could be the next electric vehicle.

I only recently learned about heat pumps and found it difficult to understand how they worked and potential benefits. Towards that end, I started hacking on this tool for others to get that info: https://www.heatpumpswork.com

[0] https://www.whitehouse.gov/briefing-room/presidential-action...

[1] https://www.theverge.com/2021/12/15/22837799/new-york-city-b...

Alteryx Innovation Labs | Engineering | Boston, MA or Remote | Full-time | https://www.alteryx.com/innovation-labs

The Alteryx Innovation Labs creates next generation technologies, tools, and products to unleash the full potential of machine learning. We’re building automation technology to make machine learning easier to use. We work on many unique and important machine learning challenges with a focus on making sure machine learning isn’t just a theoretical endeavor but has a real-world impact.

Currently, our stack is mostly Python, Scala, and React. We use AWS and docker for deployment. Above all, we prioritize the ability to choose the right approach to solve a problem.

Apply at https://www.alteryx.com/innovation-labs/careers

The post mentions this, but Compose is open source and we are continuing to improve it as we apply it to new machine learning use cases

For anyone interested in seeing more about how it works, here's the github [0] and documentation [1]

[0] https://github.com/alteryx/compose

[1] https://compose.alteryx.com/

Alteryx Innovation Labs | Engineering | Boston, MA or Remote | Full-time | https://www.alteryx.com/innovation-labs The Alteryx Innovation Labs creates next generation technologies, tools, and products to unleash the full potential of machine learning.

We’re building automation technology to make machine learning easier to use. We work on many unique and important machine learning challenges with a focus on making sure machine learning isn’t just a theoretical endeavor but has a real-world impact.

Currently, our stack is mostly Python, Scala, and React. We use AWS and docker for deployment. Above all, we prioritize the ability to choose the right approach to solve a problem.

Apply at https://www.alteryx.com/innovation-labs/careers

Alteryx Innovation Labs | Engineering | Boston, MA or Remote | Full-time | https://www.alteryx.com/innovation-labs

The Alteryx Innovation Labs creates next generation technologies, tools, and products to unleash the full potential of machine learning.

We’re building automation technology to make machine learning easier to use. We work on many unique and important machine learning challenges with a focus on making sure machine learning isn’t just a theoretical endeavor but has a real-world impact.

Currently, our stack is mostly Python, Scala, and React. We use AWS and docker for deployment. Above all, we prioritize the ability to choose the right approach to solve a problem.

Apply at https://www.alteryx.com/innovation-labs/careers

Alteryx Innovation Labs | Engineering | Boston, MA or Remote | Full-time | https://www.alteryx.com/innovation-labs

The Alteryx Innovation Labs creates next generation technologies, tools, and products to unleash the full potential of machine learning.

We’re building automation technology to make machine learning easier to use. We work on many unique and important machine learning challenges with a focus on making sure machine learning isn’t just a theoretical endeavor but has a real-world impact.

Currently, our stack is mostly Python, Scala, and React. We use AWS and docker for deployment. Above all, we prioritize the ability to choose the right approach to solve a problem.

Apply at https://www.alteryx.com/innovation-labs/careers

If anyone is looking for an open source library in this space, I work on one called Compose (https://github.com/FeatureLabs/compose).

With compose, a user defines a labeling function, and then compose scans the historical data looking for training examples to train a machine learning model.

The library has evolved as we apply it to more and more real world use cases, but it is based on approach in this paper from 2016: https://dai.lids.mit.edu/wp-content/uploads/2016/08/07796929....

Alteryx Innovation Labs | Software Engineer | Boston, MA | Full-time | On-site | https://www.alteryx.com/innovation-labs

The Alteryx Innovation Labs creates next generation technologies, tools, and products to unleash the full potential of machine learning.

We’re building automation technology to make machine learning easier to use. We work on many unique and important machine learning challenges with a focus on making sure machine learning isn’t just a theoretical endeavor but has a real-world impact.

Currently, our stack is mostly Python, Scala, and React. We use AWS and docker for deployment. Above all, we prioritize the ability to choose the right approach to solve a problem.

Apply at https://www.alteryx.com/innovation-labs/careers

This reminds me of a story I heard from Charlie Munger of Berkshire Hathaway about asking Mozart for advice.

One man came to Mozart and asked him how to write a symphony. Mozart replied, “You are too young to write a symphony.” The man said, “You were writing symphonies when you were 10 years of age, and I am 21.” Mozart said, “Yes, but I didn’t run around asking people how to do it.”