akisej

Let me ask you something I always ask my reports: if you were your manager, what would you do?

Pretty interface, although I remain unconvinced of how I'd actually use it. If I'm just prototyping for myself, LLM providers offer a decent history, and I rarely need to share notebook-style explorations of LLMs with my team. For production use cases at logicloop.com/ai we just add our prompts into code.

What's the use case you're envisioning people using AI notebooks for?

Appreciate the effort you put into this. In terms of user flow, I typically need one or two of these use cases at a time. In that case, I just type my conversion into a search engine like Google, and often use their default box.

Can you share what types of use cases you've seen people use KodyTools for?

Yup, that's part of it but I mean it bidirectionally - users can accidentally leak data to models too, which is concerning to SecOps teams without a way to monitor / auto-redact.

These common issues tend to prevent LLMs from being used in the wild: * Data Leakage * Hallucination * Prompt Injection * Toxicity

So yes it does include prompt injection, but is a bit broader. Data Leakage is one that several customers have called out, aka accidentally leakage PII to underlying models when asking them questions about your data.

I'm evaluating tools like Private AI, Arthur AI etc. but they're all fairly nascent.

Great starting point! These diagrams notably miss a LLM firewall layer, which is critical in practice to safe LLM adoption. Source: We work with thousands of users for logicloop.com/ai

How about Amplitude or Heap? For an open source alternative you could consider PostHog.

Very cool, but curious if you see people actually directly interacting with LLMs vs in a script as part of a larger application? I see myself needing debugging, visualizing output etc. so much that an IDE makes more sense to me as an interface, so want to learn about cases where that doesn't.

Aw man, sorry to hear this about your friend. Inanimate objects are directly subject to the laws of physics, but living beings that have intention and will are able to circumvent those. For example, I can jump despite gravity existing. Yes, "in the long run, we're all dead", but applying laws of entropy as a reason to not live seems indicative of a lack of will, rather than a natural law every being must follow.

As long it's clear that this is fiction, how would something like this be more damaging than a series like The Man in the High Castle, or other sci-fi that imagines an alternate universe? I think it's a nifty technique that allows us to viscerally imagine and live out our parallel universes.

Yeah, in general the more data you're able to use (assuming the context window supports it), the better results tend to be. We arrived at the data schema being a good enough compromise at which the benefits outweigh the risks for several use cases. Besides, some data stores that are generated by third-parties actually have common schemas (think Sendgrid / Hubspot activity data), so you're not risking much but potentially gaining a lot of sales ops productivity.

This seems overall well-written and well-explained, but curious for that piece on fine-tuning. This article only recommends it as a last resort. That makes sense for a casual user, but if you're a company seriously using LLMs to provide services for your customers, wouldn't the cost of training data be offset by the potential gains you have and the edge cases you might automatically cover by fine-tuning instead of trying to whack-a-mole predict every single way the prompt can fail?

Thanks for sharing how you keep the discussion quality high here. We responded to some thought-provoking questions on this thread about differentiation vs ChatGPT, other SQL Copilots, edge cases like more complicated queries etc. that we believe other HN users and makers in this space will benefit from. As I responded in @chatmasta's vouch (thank you!), we're data and software engineering experts, and happy to provide our original perspective on any other questions you have for us. Thanks for your consideration!

On simpler multi-table joins we've been able to product good results, and we've done a lot of prompt engineering to make sure it takes the schema very seriously so that prevents hallucinations too. We're always finding new edge cases and fixing those as we go.

Thanks for sharing your experience. By organizing and storing queries, we hope to be able to improve these suggestions as well behind the scenes.

Yup, our bet is that people are going to want better integration into data sources and actions resulting from their queries, and that a lot of business value will come from that.

Yes, we can actually connect to third-party apps via APIs https://docs.logicloop.com/data-sources/supported-data-sourc...

That's a great idea for future improvements, thank you for trying it out!

Appreciate the vouch, @chatmasta! I actually did write those comments myself, and think that the 3-point explanation makes for a clear explanation. My cofounder and I both started our careers as software engineers, and I've been in the data space for over a decade.

A quick Google search for LogicLoop will also show you that we're both featured on Forbes 30 Under 30, and are funded by Tier 1 Silicon Valley VCs. https://www.forbes.com/profile/logicloop/?sh=67ca2b047a89

We use a combination of APIs from existing LLM providers, and do some serious prompt engineering to get the best from them. We're starting to train models on more SQL-specific prompts now.

This feels like classic Hype Cycle content: the higher we peak on Inflated Expectations the lower we'll find ourselves on the Trough of Disillusionment.

I understand your skepticism, here's how we think about things -

* We only send data schema to underlying models: No data is actually sent there, but yes when you connect your database, we run the generated SQL on your database and store results to show those to you. They get regularly deleted though.

* We take security seriously: We're SOC2 Type 2 compliant and publish our security practices online, have a DPA, etc.

* Companies value convenience: We've often seen that companies _prefer_ cloud deploys so they don't have to DevOps themselves. Can be a matter of personal preference.

Hope that helps clarify.

Right, here's some ways we're different -

* On your data schema: A lot of the ones we've seen recently don't actually generate these queries on your schema, so you still have to do a lot of manipulation or calculation to get to SQL that just produces results.

* A better experience in productionizing SQL: Like code, writing production SQL is rarely one-shot. LogicLoop comes with a suite of AI SQL Copilot tools, like a fixer, optimizer, editor etc. that allow you to continue to iterate on your SQL logic in an integrated environment.

* Go beyond SQL: Generating query results might often be the first step of a business process. With LogicLoop, you can actually visualize results, send Slack alerts, create tickets, run automations etc. on a schedule.

Hope that helps clarify your question.

It varies. Theoretically, someone who doesn't know SQL should be able to generate it from this, but you'd still have to get someone to connect your datasource, and have a general sense for what you want to do.

I'd say it's not optimized for the high-level executive or entry-level operations person. It's more geared for someone that is data-aware but wouldn't consider themselves confident in SQL, like a junior data analyst profile.

Good question. A few different ways:

* Model Selection: We pre-select the best models for SQL generation, so you don't have to A/B test and figure it out. As the number of models available increases, this can take a lot of time if you want to do it yourself.

* Prompt Engineering: Similarly, we've had to fiddle around with a lot of different prompts to deal with common issues like hallucinations (and are constantly updating them, fixing bugs, telling the model what it can't ignore etc.) This also takes a lot of time, patience and skill.

* Directly on your data schema: You'd have to copy-paste your schema every single time you wanted to generate or modify a SQL query, which can be cumbersome. You connect your data sources once with LogicLoop, and it gets auto-included correctly for you.

Hope that helps.

Thank you, let us know what you think when you try it out.

Agreed, that's always a risk with making data too easy to access, its context doesn't always come with it. While we initially thought it'd be most valuable for new SQL users, a lot of existing SQL experts actually find that it gives them a huge productivity boost.

Haha, almost :P Some problems can't be solved. For everything else, there's LogicLoop.

Yeah, it's actually pretty good at fixing existing queries if you give it your schema. I'd say on queries from 10-50 lines of SQL it works 9 out of 10 times.

Thanks! We're starting with just the schema, good clarification.