HN user
Prophecy.io | Think Cursor for Data Analysts | US (onsite SF, remote) Prophecy is used by data engineers in the largest enterprises (SeriesB, $120M raised). Our new product for data analysts is agentic and multi-modal (docs, low-code, code). - AI Research Engineer - AI Agentic Software Engineer Looking for top tier AI engineers who are excited to ship products at tremendous pace email me - raj at prophecy dot io
Prophecy.io will be the way everyone does data transformation soon, just a zero compromise solution that beats legacy ETL and code by miles
Yes, but in addition to being ACID compliant (serializable ANSI), you need to support a million transactions per second - it’s not just about data size.
Why is this just not ETL, why do you need anything here? There is no new category or product needed here.
That’s assuming you’re not using low-code. There are inbuilt connectors to read data, transform data, read/write to pinecone, make api calls to LLMs. It is much faster to prototype with Prophecy.io
It is pointless - LlamaIndex and LangChain are re-inventing ETL - why use them when you have robust technology already?
1. You ETL your documents into a vector database - you run this pipeline everyday to keep it up to date. You can run scalable, robust pipelines on Spark for this.
2. You have a streaming inference pipeline that has components that make API calls (agents) and between them transform data. This is Spark streaming.
Prophecy is working with large enterprises to implement generative AI use cases, but they don’t talk so much on HN.
Here’s our talk from Data+AI Summit: Build a Generative AI App on Enterprise Data in 13 Minutes https://www.youtube.com/watch?v=1exLfT-b-GM
Here’s a blog/demo https://www.prophecy.io/blog/prophecy-generative-ai-platform...
Yes but this is just ETL - LlamaIndex and LangChain are re-inventing it - why use them when you have robust technology already?
1. You ETL your documents into a vector database - you run this pipeline everyday to keep it up to date. You can run scalable, robust pipelines on Spark for this.
2. You have a streaming inference pipeline that has components that make API calls (agents) and between them transform data. This is Spark streaming.
Prophecy is working with large enterprises to implement generative AI use cases, but they don’t talk so much on HN. Here’s our talk from Data+AI Summit:
Build a Generative AI App on Enterprise Data in 13 Minutes
https://www.youtube.com/watch?v=1exLfT-b-GM
Here’s a blog/demo
https://www.prophecy.io/blog/prophecy-generative-ai-platform...
A lot of B2B startups can technically the cloud API to provide value added applications to Enterprises, but often the banks and healthcare companies will not want their data running through startups pipes to OpenAI pipes.
We provide a low code data transformation product (prophecy.io), and we’ll never close sales at any volume, if we have a to get an MSA that approves this. Might get easier if we become large :)
Prophecy (prophecy.io) | ML Engineer | Palo Alto, CA
Work on LLMs and Knowledge Graphs fir English -> ETL data pipelines
You should have previous experience in NLP and solid programming. Shipping a project from scratch is important.
If you have only worked at large companies on tuning a small cog, it will not work
reach out to raj@
are you serious? they invested in US Treasuries, the highest rated and the most conservative asset class.
Banks are in the business of lending money to home buyers, etc - how is loaning to the US Govt. criminal?
They have a cash crunch due to interest rate rise, and VC slowdown - bad risk management, but everything is above board.
Startup lines of credit should not happen from deposits, too risky an asset class
Prophecy.io | Software Architect | Palo Alto, CA
Expert in Spark, Spark internals or Expert in Query Compilers.
Level variable (2-20 yrs exp), prefer 10+ years
Prophecy.io | Data Engineer | Palo Alto, CA
Expert in writing Spark code and running Spark at scale with substantial production and performance improvement experience. Work with our customers and help product design.
Level variable (2-10 yrs exp), prefer 5+ years
send mail to raj@prophecy.io
Prophecy.io | Customer Success Director | Palo Alto, CA
We’re a game changing low-code data engineering platform, replacing ETL tools and modernizing the modern data stack.
You know data engineering well, have excellent customer facing experience (consulting works). You will work with our enterprise customers, coordinating with our strong data engineers to make the customers successful. Be their trusted advisor, no upsell responsibility.
email: raj@prophecy.io
Its not public language, but at Risk Management Systems, I worked on Contract Definition Language- an external DSL for computable insurance, re-insurance contracts. This was used to simulate insurance losses when catastrophic events happen (earthquake, hurricane)
Thank god solving customer problems has taken over the tech-ceo job, starting with tech is usually a bad idea, even when it is innovative (crypto-ponzi as an example)
Prophecy.io | Customer Success Lead | InPerson Palo Alto We have customers from midsize to Fortune 500 relying on us for Data Engineering. We’re building out our success organization to onboard, enable and guide the customers. We do have a top notch data engineers in the team, need leadership. Level is flexible.
send message to raj @ prophecy.io, only primary candidates, thanks!
Prophecy.io is hiring a technical content writer in our marketing org. Requires data engineering, spark knowledge, good communication and writing skills and desire to move toward product marketing - the most in demand skill right now! Based out of our Palo Alto office, full time, good compensation, benefits.
send mail to raj @ prophecy.io
We’re very different from the ETL tools in that we’re bring software development best practices to data.
When you do visual drag and drop - prophecy is generating high quality code on git that is 100% open source (spark, airflow), you have tests and CI/CD - so you’re visually doing solid data engineering.
You can toggle between code and visual - so if you change the code (some), the visual graph updates - so small edits directly to git don’t break the visual layer.
All visual components are generated from a spec - think a Spark function with a some more info. So the data platform teams will create their own library/framework and roll it out to the wider teams. How it works us that in the visual editor, you start with standard Spark library, but can load visual components for delta, or encryption or data quality.
Our customers are typically fed up of these ETL tools and moving to us. We can also import the ETL formats (AbInitio, Informatica, …) in an automated way (we reverse engineered their formats and created source to source compilers)
Prophecy.io let’s you create visual components from any Spark function. Same with Airflow. So you can use standard components (built-in or your new ones) without being restricted.
Founder here - we’re working to solve this exact problem.
Our US based startup has engineering in India (20 engineers) and the biggest challenge is finding engineering leadership.
It is hard to hire in the bay area, especially leaders who have experience shipping high quality products (systems), so we built the team in India.
Many young companies like us struggle since a lot of the talent (definitely in the bay area) is locked up in the Faangs where poor fiscal/monetary policy in the US has inflated stock part of the compensation so much, that it makes no sense to leave.
So it is hard find principal engineers to do great work because they choose to go to large companies and get poor work for money.
The bar is high for us though, we’d look for engineers with specialized skills (compiler, database internals) and leadership on complex products. The middle is the dead zone (engineers with N years of experience wiring apps)
Not sure about the European market, but maybe you can try to go to a smaller company?
Prophecy.io | Principal UX Designer | Palo Alto, CA ONSITE | Full-time
Prophecy is a low-code data engineering product that provides a visual environment to design, deploy and manage data pipelines.
We’re looking for our first UX designer. Our product is quite complex, so we’re thinking a degree in engineering (perhaps something similar to Stanford CS HCI specialization), and UX experience in data or visualization space will be a good fit.
We pay well and have a great team - we’re friendly and everyone is very good at what they do.
If you think you might be good at this, just reach out to me at raj@prophecy.io
We work with many businesses that are larger (Fortune 500) and the T per pipeline is say 60 steps with 1200 columns at 10TB scale and uses multiple things not in SQL. They lookup object stores, lookup web services, use rocksdb, partitioning is important. At scale, cost becomes critical- some are even moving to their own Spark on Kubernetes. ML on done on data after ETL into Data Lake.
None of them can use DBT for core ETL, but DBT might be good later for views, some dimensional modeling. They have done a good job here.
Think of it as the modern small-scale data stack.
First, product-marketing is the most important skill in forming a company. You might get lucky without it, but success rate will be low.
This post is good for learning, we have too many “systems“ developed in Java (Hadoop et al - I’m looking at you - you sluggish bugs in the rug), we need more programmers to learn C and build systems in it.
Also, you can do this so much easier in scala using parser combinators, if the goal is time to market and the code to be compiled is not very large. Here is an example to see how the approaches compare: https://www.prophecy.io/blogs/scala-packrat-parser-combinato...
These are not flaws, but tradeoffs made by deliberate choice.
I’ve worked on x64 and graphics compilers and this article should be skipped.
We want software engineers who understand compilers and query optimizers and systems.
This requires good theoretical foundations with solid engineering skills.
It is extremely hard to get these. Most CS majors don’t know much CS - top grads from top US schools undergrads are clueless usually.
That’s why you get projects like Hadoop- a large scale systems project written in Java instead of systems languages - because the engineers are poor quality in top companies as well.
Prophecy.io | Spark Solutions Engineer | US - Remote (some client travel) | $180k-$220k
Represent Prophecy at our Fortune 500 customers. Help them migrate from legacy ETL products to Prophecy on Spark. Help them on-board and solve various use cases with Prophecy. Channel product feedback back - be a bridge between customers and our engineering talking to both everyday.
Travel customer sites occasionally - build relationships.
Hands on experience with ETL using Spark is required Experience in customer facing role required.
https://angel.co/company/prophecy-io/jobs/1528717-senior-sol...
Our industry is doing quite well (If you leave the ad-economy out - basically Google/Facebook)
Cloud computing, Open Source Software in commercially feasible way and so on. Tremendous growth and innovation is underway.
We do have a problem of wokeness, left in SF destroying the city, teacher unions hurting competitiveness of kids, police unions stopping all police reform in California.
Basically just like USSR, all the do-gooders - claiming to do good in the name of the “people” are the problem.
So, don’t be a protester - do real work, build something, be the solution. Create a small open source project of value. This is really really hard - that’s why do many people protest instead - that is easy.
I assume SaaS=SaaS-App like iPhone App. Because you can also build serious technology as SaaS.
We’re building a primarily PL startup at Prophecy.io - Low Code Data Engineering Platform. We user parser combinators heavily, code generation and all. Hard to find skilled engineers these days in compilers though.
The Low-Code movement seems very focused on programming, no?