HN user
Really awesome seeing more people work on this! I’m one of the founders of Opik https://github.com/comet-ml/opik which does similar things but also has a UI and supports massive scale. Curious to hear if you have any feedback!
Excited to see more people building in this space. From what we've seen with customers it's critical to be able to compare what you're seeing in production to what you trained on (rather than historical period). That's almost the textbook definition of drift. Do you have a sense on how to approach that?
At Comet.com (disclaimer: i'm the CEO/Co-founder) we provide experiment tracking and artifacts management so we have the training distributions for comparison. I'm always curious how it looks like for a monitoring only solution
Hi HN! Excited to share with you our recent open source project CodeCarbon - a collaboration between Mila, BCG GAMMA, Haverford College, and Comet.ml. CodeCarbon records the amount of power being used by the underlying infrastructure from major cloud providers and privately hosted on-premises data centers. Based on publicly available data sources, it estimates the amount of CO₂ emissions produced by referring to the carbon intensity from the energy mix of the electric grid to which the hardware is connected.
Would love to hear your thoughts!
Github: https://github.com/mlco2/codecarbon
Blog post: https://www.comet.ml/site/introducing-codecarbon-an-open-sou...
Demo (hosted on Comet.ml): https://www.comet.ml/mlco2/codecarbon-test/view/XFNDEoLvxysp...
Hey! I'm one of the founders at Comet.ml. We believe that Git should continue to be the approach for managing code (similar to dvc) but we adapted it to the ML workflow. Our approach is to compute a git patch on every run so later you can 'git apply' if you'd like (https://www.comet.ml/docs/user-interface/#the-reproduce-butt...).
Streamlit is awesome but solves a different use case. There's definitely similarities but also a few core differences. 1. Comet Panels reads the experimentation data from our API/Database. Streamlit is much more like Jupyter notebooks where you'd have to setup your own data store and implement logic to update it with new experiment data. This also means that if you pull a visualization from the gallery it should work out of the box as the API is standard among everyone! 2. Comet Panels is freely hosted so no need to setup a webserver 3. Comet Panels is natively written in JS/HTML/CSS so you can use any JS library out of the box. Streamlit is written in Python and you might be limited to the built in UI components (or you can build a custom on which is also in JS/HTML/CSS)
Hi HN! Really excited to share this with with those of you training ML models. Comet Panels is an easy and free way to build visualizations, widgets and apps in HTML/JS/CSS on top of your ML experiment data and a gallery where you can share your work with the world.
Panels is built on our experiment management platform (100% free for individuals). Essentially as a user you can report all of your metrics, params, assets, data samples etc. Everything stored is also accessible via our JS SDK so it's really easy to fetch data for your panels.
The main idea is that visualizations aren't stale. Every time a new experiments is reported your visualization will update with that additional data.
Docs available here: https://www.comet.ml/docs/javascript-sdk/getting-started/
Blogpost with more information: https://www.comet.ml/site/introducing-panels-custom-visualiz...
Also keep in mind that unlike tensorboard we keep your full data series available in the API and only downsample the charts to 15k points.
M0zg do you mind sending me an email with your project? Happy to look into it. gideon a t comet.ml
just to clarify - are you referring to this experiment? https://www.comet.ml/demo/urbansound8k/be09e32700cd435fb6b55...
we're working on another version fixing the folds issue on Urbansound8k and will update the article asap.
Hi M0zg! Gideon from Comet- sorry to hear you're having issues. Did you every try to report these? if you share more info at support@comet.ml or at our slack channel i'm sure we can fix it / improve. On a general note: 1. you can see min/max values in the metrics tab for finished/running experiments. 2. we spend tons of time on performance but these are actually difficult problems to solve, i.e if you have ten charts all showing 10k data points all updating in real time. That said if you share your project we can use it to improve. Finally the SDK is designed to never crash or slow down your training and this is the first time we've heard that complaint - again please ping us so we can figure out what's going on.
Really excited to see this relaunched. Every DS team has issues around dataset management. We previously shared a tutorial on how to get a fully reproducible pipeline with Quilt + Comet.ml https://blog.quiltdata.com/building-a-fully-reproducible-mac...
Comet.ml - multiple engineering positions | New York City | Full-time | Onsite (NYC) or REMOTE
About us: Comet is doing for ML what Github did for code. We allow data science teams to automatically track their datasets, code changes, experimentation history, and production models — creating efficiency, transparency, and reproducibility. Ever wondered how Linus felt when he invented Git? How about the brains behind JIRA?
Join us to help write the future of ML workflows and teams.
We're hiring for:
-Senior Front-end Engineer: https://www.comet.ml/careers/swe-fed
-Senior Backend Engineer: https://www.comet.ml/careers/swe-backend
-Senior Full Stack Engineer: https://www.comet.ml/careers/swe-full
-Marketing Associate https://www.comet.ml/careers/marketing
Check out the team and investors here: https://www.comet.ml/about
A common error is target leaking. An AutoML system will likely consider this a "strong feature". This is where having someone that actually understands the business domain is critical.
There's no question that there's value in AutoML system yet most ML production systems I've worked on / seen were way more complex than feature vector -> model -> prediction. You likely have multiple models, pipelines, normalizations and plain old conditionals. Hard to automate all of this.
Comet.ml - multiple engineering positions | New York City | Full-time | Onsite (NYC) or REMOTE
www.comet.ml
About us: Comet is doing for ML what Github did for code. We allow data science teams to automatically track their datasets, code changes, experimentation history, and production models — creating efficiency, transparency, and reproducibility. Ever wondered how Linus felt when he invented Git? How about the brains behind JIRA?
Join us to help write the future of ML workflows and teams.
We're hiring for:
-Senior Front-end Engineer: https://www.comet.ml/careers/swe-fed
-Senior Backend Engineer: https://www.comet.ml/careers/swe-backend
-Senior Full Stack Engineer: https://www.comet.ml/careers/swe-full
Check out the team and investors here: https://www.comet.ml/about
Sure you can reuse tools to achieve similar results. As with everything else the devil is in the details. Does your monitoring system saves results forever or it only let you report 90 days back? Can you compare two runs in a meaningful way? i.e not just logs but also interactively plotting exploring your results? Do you need to spend hours to instrument your code? Can you sort jenkins job by a parameter/metric? What about reporting new results to an existing experiment? There's many more examples. But in any case if you can reuse your CI/CD system for ML experiment management you should do that. Another question worth considering is that if this is a "failed idea" why would engineering led tech companies build these systems? Obviously they tried reusing their current tooling.
The tools we've been building for the past fifty years were designed for software engineering. Machine learning workflows are different in many ways and as such require new tools and approachs. That's at least our perspective.
Runs and experiments are not 1:1 mapped. A single container run could generate multiple experiments such as with the case of parameter search. Additionaly traditional tools for version control are not well suited for ML results and exploration. That said code is still a big piece of the puzzle. Our approach at Comet.ml is to snapshot everything whether it runs on a container or not and tie that back to git.
Comet.ml | ONSITE | New York, NY | Senior Backend Engineer | Full-time | VISA | Comet.ml is doing for Machine Learning what GitHub did for code. We allow data science teams to automagically track their datasets, code changes, experimentation history and production models creating efficiency, transparency, and reproducibility.
Ever wondered how Linus felt when he invented GIT? How about the brains behind JIRA? Machine Learning teams operate like software teams 15 years ago and we started Comet.ml to write the rules of ML workflows.
https://www.comet.ml/careers/swe8
We're looking for a senior backend engineer to join our team of 8. Comet.ml is built using microservices (Java/Python), Redis, Kafka, MySQL, NGINX. We are funded by Amazon, TwoSigma, Trilogy and others.
https://techcrunch.com/2018/04/05/cometml-wants-to-do-for-ma...
careers@comet.ml
My biggest issue with notebooks is that we're throwing aways years of best practices. Notebooks often lead to untested code with poor structure, frequent use of global variables and readability issues. Even using it as debugger is limited since there's no way to step into functions.
I was mostly referring to coupling code with results. For example you have a code that loads a dataset from S3 and then trains a neural network. If you only use git you're likely to lose the hyperparms info (which is often passed as command line arguments) and your metrics/results.
We actually do code and model versioning (and simple data versioning). One thing to keep in mind is that code/results/hyperparams must be coupled. If you have a git branch with some training code and you do not know what the hyperparams/results are then it's not very valuable.
Since we do not host your data we cannot provide filtering on the actual dataset content. We do allow you to track where the data was coming from and if it changed (by hash). Same for checkpoints, you can log their location (S3/local path) and hash.
This article seems to discuss genetic algorithms which could be used for hyperparam optimization. We use another method called Bayesian (GP) hyperparam optimization. According to our internal benchmarks and academic research Bayesian methods outperform genetic algorithms. Another thing to keep in mind that we automate the entire process for you. You only need to provide a list of parameters you'd like to tune.
No. You'll be able to see both runs with the code diffs.
It depends if it's a git project but pretty much yes.
Both CometML and Tensorboard help track metrics/weights during training in a similar way. CometML also tracks your hyperparams, code, dependencies. We also allow you to compare models, collaborate by sharing projects and experiments and much more. TB is an amazing tool but it doesn't help with reproducibility.
If you're already using Tensorboard just throw in our one liner: comet_ml.experiment(api_key="your-key") and you'll get everything TB gives you + our added value.
Thanks @jorgemf. Keep in mind that $745 also includes unlimited usage of our hyper-parameter optimization service.
Sure. Shoot us an email and we'll get you started! mail@comet.ml
We do not store your data as it's usually a very sensitive. You can host your data on GCloud or AWS and use Comet.ml to track where it was coming from and if it changed between experiments.
Thanks! We indeed solve a similar pain point as Domino but we unlike them we allow you to train your models on your own infra/laptop.
As for monitoring production models that's something we're also working on. It was important to get the training part out first so we can measure those distributions changing.