clausok

Kx released a 32bit, free-for-commercial-use version in 2014 and then reversed course around a year later after banks and hedge funds surprised them by flocking to it for a large subset of their developers / dev machines.

Hopefully they stick to it this time around. It's an incredible system. It's the only thing I've ever used, including Pandas, dplyr & Matlab, where someone could stand over my shoulder asking data analysis questions and I could answer them, on the fly.

LLM's, though notoriously bad (so far) at KDB+/Q compared to other languages, are still a godsend for folks getting started. I recently returned to writing Q after being away from it for 8 years and I've been amazed how good even Google's AI suggestions have been at helping with functions & queries.

Getting started tip: try using Q strictly as a query language, avoid K. Do everything else (data shoveling, devops,...) with a different language.

I'm always stunned to see the level of VB6 expertise that remains in the world even two decades after Microsoft left it for dead.

Just look at the effort and knowhow that went into this VBA function that resolves the local file system path from the https url of workbooks synced to OneDrive/SharePoint:

https://gist.github.com/guwidoe/038398b6be1b16c458365716a921...

I've been surprised to see many pro devs using Excel/VBA as a secondary tool.

One example: a couple years ago I was working with a big hedge fund and one of their data analysts sent me an Excel model he had built and I was tickled to see the .xlsm extension (i.e., VBA code on board).

"Ahh ha", I thought, "Let's see what these macro-recording cowboys have been up to."

There was a lot of VBA inside, all written by this Caltech comp sci data analyst who was a Python superstar. The VBA was for pulling data from a database, putting it on a sheet, building some formulas, and some pretty formatting. There were even a few userforms!

I teased him, "VBA? What else are you guys using over there? A cotton gin and a steam shovel?"

I was startled to hear him heap praise upon Excel and VBA instead of the usual complaints.

He said something that stuck with me, "Excel makes it easy to understand the dependency structure that is implied by computations. If I had done this in Python, I'd be answering questions about it all day long."

He has a younger sister. As for other outlets for making friends and socializing, team sports are indispensable. Also great, and surprisingly underutilized -- in my observation -- are playgrounds.

With playgrounds you get the best part of school, recess, without the rest of it.

If parents would look up from their phones for a minute and watch what's going on in playgrounds they'd be amazed.

Recently I saw about 20 kids, across a wide age range, hold a spontaneous, well-organized game of Among Us.

You won't just make friends there; you'll run in Draco Malfoys too; and that's also valuable.

We started homeschooling by happenstance when it turned out that kindergarten was going to conflict with our son's beloved jiu-jitsu class. The class was a bit of a drive and we weren't going to be able to get there in time for his favorite part: the 30 minutes of horsing around on the mats before class started.

We thought, "Well, it's only kindergarten". It worked out well enough (knock on wood) that we just kept going.

Compared to my own harried "THE BUS IS COMING!!!" sleep-deprived school days, it has been a wonderful change.

If I could go back in time, I'd put my mom in an ankle lock until she agreed to do the same for me.

Especially since I learned that in my home state homeschoolers can participate in high-school sports. In high school I had to wake up at 4AM for 5AM hockey practice. If I could have returned home afterwards and slept, rather than going straight to school, I would have thanked the Gods.

I agree that insider trading is an unlikely explanation for the reason you mention.

It could be that an early algorithmic advantage compounded into an incumbency advantage: better research tools; better data; better pay for better scientists.

The 'secret sauce' may have started out as novel discoveries and, over time, become 'just being RenTec'.

I have been consulting at a world-renown hedge fund that has a lot of F# including nightly processes that have apparently been running for years with zero maintenance. You'll never hear them talk about it or see them at meet-ups though. The CTO is a bit of a maverick -- I'm not suggesting that this type of quiet adoption is common.

If you use collection classes in vba you can run into this bug pretty quickly. Last year I deployed an Excel add-in to a client that enabled them to run Snowflake queries concurrently from vba and this line:

    If queries.FilterHasError().Count > 0 Then

I had developed the add-in using 32-bit Excel. This was an extremely confounding troubleshooting experience. Eventually Google & Stackoverflow rescued me from believing I was going crazy.

This debate about whether to use named-ranges in Excel formulas brings back some distant memories. I was on a team at Goldman Sachs that built Excel tools for the investment bankers and because of that we'd help out with the new banker Excel training held at the giant Chelsea Piers sports complex.

Every year freshly minted MBA's would begin the training and immediately become aghast when the trainers told them to never, ever, use named-ranges in formulas. Not only had the trainees been explicitly taught to use named-ranged in their MBA programs, but any idiot could plainly see that [=enterprise_value/ebitda] was better in every way than [=C13/F22]. More expressive, more readable, easier to spot errors, etc.

The trainers would argue that in an MBA program you build your models, submit them, and move on to the next assignment, so you don't get a view of the longer-term problems that arise from named-ranges. What does [=ebitda] actually tell you? Is that the last quarter's actual number, is it the current estimate of the next fiscal quarter, is it a 12-month blended forward estimate?

And as you try to solve those questions with [=ebitda_est_next_fiscal_qtr] you often end up with two more problems: an unambiguous name for you can unambiguously mean something else to someone else and as you update your model over time, if you forget to update your named-range references, you have formulas that look right, but are wrong, e.g., [=ebitda_est_next_fiscal_qtr] now refers to an actual reported ebitda number, not next quarter's estimate.

[=C47] while not telling you much at all, is unambiguous. It doesn't look "right" or "wrong". It can't mislead you. If you want to know what it is, go look at cell C47. The new MBA's would argue, "Wait a second, C47 could now point to the wrong data as well, and it could be mislabeled with a stale row or column header. And the trainers would reply, "Exactly, and when you go to see what C47 represents you will have the context to recognize those errors and fix them."

I encountered exactly this when my brother went to film school. The students would rotate roles and it was his turn to be the producer. One day I came home from work and he was having problems with his Excel-based budget sheet. I noticed that his row of sums at the bottom of one sheet had hard-coded values rather than formulas. Thinking he had inadvertently pasted over them, I told him, "Look, you killed your formulas." "What are formulas?", he asked. Turns out that he had been tallying up the columns with a calculator and then typing those totals into Excel.

Same for me. My high school and college hockey coaches could really let you have it. They never pulled you aside and did it in private either. My college coach had episodes that would make even Bobby Knight look like a pussycat. He once had a roll-on-the-floor grappling fight with a teammate in the locker room between periods. (Coach had a big tactical advantage: he wasn't wearing skates) When I ended up working on a trading desk the impromptu performance review broadsides -- in front of everyone -- felt very familiar.

More color on the original FoxPro matching engine:

"Levine found a simple but hugely consequential 'trick' to speed matching. In Levine’s later paraphrase, what the 'enter2order' procedure did when the Island system received a new order was to: 'See if there was a record from a recently cancelled order that we can reuse for this new order. This is hugely important because that record will likely still be in the cache [fast internal memory] and using it will be much faster than making a new one. (Levine, n.d.)'"

http://www.sps.ed.ac.uk/__data/assets/pdf_file/0003/97500/Is...

I agree on the key verticals approach as being the best bet. Perhaps the best way to dip your cup into the raging river of Bloomberg cash flow is to take a page out of the Stripe/Twilio book and focus on API first, with phenomenal Excel and Python interfaces, docs and examples. Just forget about a terminal-like UI, or any UI at all and build the tools that make it easier for other people to build their own niche apps. Money.net, Sentieo, YCharts, Koyfin -- my observation is that these all treat the programmability of their data and analytics as a secondary focus, or not a focus at all. It would be nice to see someone come along and do it the other way round.

I've been waiting 15 years for litigation in this arena. If prices, like player statistics from an NHL game, are historical facts which can not be copyrighted, why don't we have better and cheaper public market data API's by now? How have the exchanges been able to maintain a stranglehold over distribution of historical facts?

Speaking of this breadth, a few years ago I looked at Bloomberg's PORT suite of portfolio performance attribution and risk management functions to see what they offered and how programmable it was. On top of the usual extensive help docs, they had 56 white papers totaling ~1400 pages (as of June 2017) explaining the calculations.

Just like you say, I passed the AP calculus exam in high school and then when I arrived at college my advisor, who also happened to teach a freshman Calculus 1 section, told me, "I know you passed out of it, but why don't you sit in on a couple of my classes and see what you think?" I ended up staying the whole semester when it became clear that I didn't understand anything about Calculus.

The desk-aligned dev spot is a tough one to land. If you’re not hailing from CalTech, MIT, et al, I suggest the highest-probability strategy is an indirect one: get a client-facing job at hedge fund administrator, get great at Excel+vba; over-deliver for the fund’s CFO and/or traders in all their post-trade questions (that’s why Excel+vba expertise is important: it’s easy to deploy to them as just an Excel file); get recruited. The right spot at a fund administrator is much less competitive, and you get an opportunity to impress the same people you would if you were hired directly onto a desk.

Excel+vba is an odd beast because, nowadays, it's rarely seen as a differentiating skill -- it's not taken seriously -- and yet on almost any trading desk there is an infinite series of todo's where Excel+vba is just the right tool for the job.

One thing to consider, which I've found powerful, is a hybrid approach where you use vba for what it's best at -- interacting with the Excel object model -- and have the vba load and call a C# xll that leverages the .Net libraries and does all the work that would be tedious to do in vba. For your users, they would simply open a "host" .xlam file, which is the only one they would be aware of, and this file would load additional .xlam or xll files and typically also check for updates to these files. No installer necessary.

For example, your vba code would pick up a range of index (or ETF) tickers from a sheet, use Application.Run("lib.constituents") to call a C# function that pulls the stock tickers and weights that represent each indices' constituents, your vba code would receive back an array of arrays and use Excel's row grouping feature to group the members & weights under the parent index and perhaps add some formatting and\or formulas.

The advantage of this division of labor is that you get faster dev & debug feedback loops within your Excel-centric code, while minimizing the amount of time you spend working in a dev environment that has stood still for 20+ years as others have raced far past.

Your Excel+vba application’s features remind me of when I joined an investment bank in 2000. I had come from an insurance company where I was considered the Excel\vba wizard, and I was impressed, in the extreme, by my new colleagues’ approach to Excel development. Even during the interview process I had realized that, when it came to vba, I was but a babe in the woods. They had auto-updating code (when you “published” code, all client workbooks downloaded the latest version), code-generated collection classes, interface inheritance, tests, error detection, higher-level functions via Application.Run(), self-contained worksheets with embedded vba code that would operate even if moved to a new workbook.

Inevitably, new hires would be unhappy that they did, in fact, have to write vba code all day and would argue for switching to a better language. Our team manager would say, "Vba gives us a superpower no other language does: we can deploy whatever we want, whenever we want, to whomever we want. In any other language, getting 'Hello World' in front of a user is a six month project."

Good point. I've seen miracles occur simply by refactoring an Excel workbook such that the inputs, outputs, and calculations to map the inputs to the outputs were all delineated into their own clearly-formatted bounded context. And also, when you do drop into vba code, simply organizing the code so that interactions with the sheet are easier to understand, and isolated -- and focusing on "seriality" (no feedback loops, minimize code triggered by events) can transmogrify vba code from the Bogeyman into a friendly neighbor.

Plus, in my observation, there is often a dismissive attitude towards Excel among the generic developers which blinds them to the risk of this “re-implementing Excel’s features” cycle.

I worked on a team that helped business users who had "outgrown excel", i.e., they had hung themselves with the rope Excel provides. Almost always their scaling problems were solved simply by better Excel practices: better management of the calculation mode; setting the RTD throttle interval to 1-2 seconds; replacing Bloomberg's streaming data function (BDP), with the native alternative {=RTD("BLOOMBERG.RTD...)}; optimizing the division of labor between what is done on the sheet with formulas versus with vba\xll code; meta programming, i.e., creating all or part of your calc sheets and formulas with code so that you get the understandability and observability of Excel formulas while avoiding the things that are hard to do with formulas, such as grouping, joining, filtering, looping; making 3rd-party add-ins workbook-specific such that they're not always on (many of these add-ins listen to application-level events like selection_change and on_calculation which can diminish the performance of all open Excel models, even the ones that don't use that add-in).

I interviewed with a quant hedge fund in 2004 whose systems where all in FoxPro. They loved it. The general partner of the fund believed that Microsoft had figured out that they could unbundle FoxPro into 3+ components (DB, IDE, Reports...) and sell each separately. It's been a mystery to me why no one, using sqlite as the DB foundation, has attempted a modern-day FoxPro\MsAccess-like tool.

Agree. If you're selective about which bits and pieces you move away from Excel, and not religious about getting rid of Excel entirely, you can often have a great result.

What trips people up is that Excel models can be deceptively inviting of replacement ideas. They may have many workflows within them that are obviously much easier to accomplish with a different tool. E.g., grouping and applying aggregate functions; filtering; joining; filling missing data, especially with constraints such as filling forward EPS estimates but only into the same fiscal quarter; etc. You could look at an ocean of INDEX(,MATCH(...)) functions in Excel and discover that the same manipulation in Pandas would require just two lines of code. But it is easy to overlook the importance of seemingly simple calculation flows, in the same model, that are uniquely suited for being expressed in Excel, the universal language for describing calculations.

For example, I have seen data analysts move from Excel to Python, replacing their models as they go -- after being wowed by Pandas' data manipulation capabilities -- and then get bogged down for months trying to recreate what Excel's =RTD("BLOOMBERG.RTD","",...) already did for them. And when the portfolio manager tells them, "this number looks wrong", which happens often, they spend half a day dumping data into Excel and building a sheet that's illustrative for the PM.

I know what that voodoo may be. Of the many failed "replace Excel" projects I've seen in the investment banking \ hedge fund world over 20 years, the usual mistake is that the focus was on the specific Excel model itself and not the generic building capabilities that Excel provides. An Excel model, i.e., the specific map from inputs to outputs, is typically fairly easy to replace with another tool. But what you've done is replace just the pants, not the sewing machine. Excel is the pants AND the sewing machine. And that sewing machine part is very hard to replace. So you can end up with the same model in C++, the very same map from inputs to outputs, but the user's ability to make changes and understand the model is greatly diminished. So they say "no thanks" and stick with the Excel model.

The worst Excel user I've ever seen was my brother. At film school he was producing a student film. He did the budget in Excel. One day I came home from work and he had some questions waiting for me. Looking through his sheet, I noticed that his summation row at the bottom had hard-coded values rather than the =SUM(...) formulas I was expecting. Thinking he had inadvertently done a 'copy -> paste values', I said, "Look, you killed your formulas here." He asked, "What are formulas?". Turns out, he had been using a calculator to tally up all the columns and was typing the total into the cell at the bottom.

Maybe not the same thing, but Excel's row scrolling with the arrow keys was driving me nuts until I discovered this setting: Options -> Advanced -> Display -> 'Disable hardware graphics acceleration'. After I checked that box to disable it, scrolling was much better.

Adding my data point: as a user of the terminal since 2005 I don't ever remember it crashing. Regarding the API (.Net) the only trouble I've had was when I was using a BlockingCollection to keep my simultaneous requests under the Desktop API limit of 128 and I still received a "too many requests" error -- but I trust Bloomberg's throttling software more than my own so I didn't complain.

re WPF vs winforms: consensus recommendation would be WPF. But for something you're building for yourself, WPF has a lot of complexity that you may not get much benefit from. If you decide on WPF, don't use xaml! Separating dynamic UI into a static markup language is a complexity disaster that will cost you a lot of time. (Note that google results are dominated by xaml answers, so if you want code-only answers it's best to search github.)

Best approach may be a third option, at least as a prototype. Use Excel, with a RTD server that connects to TT. Uninstall the Bloomberg Excel Tools add-in and use direct calls into BBG's own RTD server, i.e., =RTD("BLOOMBERG.RTD","","VGA INDEX", "LAST_PRICE") instead of BDP(...) for market data (~100x faster), and you can get to an 80% solution very quickly.

re F# vs C#, I had used C# for a decade before switching (after I had a vacation ruined because of a multi-threading bug in our position-sizing model. Multi-core parallelism is much easier in F# due to default immutability). If you play around with the F# REPL you'll pretty quickly find out whether the language fits your mind well or not. For me, as soon as I read this code (http://fssnip.net/5I), I knew I had found my native language.

As someone who used xtrader a lot in the past, I'm a little sad to see it get claimed by the web-based movement. That was some really fine software.

To get a sense of the tradeoffs taking place that leave you with a noticeably slower UI, see here: https://www.forbes.com/sites/tomgroenfeldt/2019/02/12/openfi...

PSA: It may be a good idea to hire a programmer and customize your software. TT has a nice API. If you have idiosyncratic workflows or views, or if you just want a few particular things to be faster, it's not too hard to use this API to make the tradeoffs and optimizations that work best for your own supplemental UI screens. We did this with a F# + WPF UI where the fill feed and responsiveness was just as snappy as TT's own UI (the old, c++ one).

We also found using Excel as an output to be a powerful design pattern. Excel is a great language for explaining the flow of calculations. We used to have WPF screens where a portfolio manager / trader / analyst could right-click on a number and select "show details" and he/she would get an Excel workbook where that same number would be selected and they could see the derivation by following the cell's precedents. It was not always feasible to show a complete derivation, but we'd try to show a useful amount of elaboration. This was a heavily used and loved feature, often leading to the users making edits or additions to these detail sheets and giving it back to us as a spec. Building the 'details' Excel books programmatically was a lot of work, but part of the payoff was that we also used them as a nightly check on our primary system, i.e., we'd automatically run 'show details' on everything and make sure the 'detail' result matches the primary system result. That comparison caught many bugs. Kent Beck: "More important than tests vs types is the principle of double checking. If you say something twice in independently derived ways, you're more likely to be correct than if you just say it once. Tests are a form of double checking. So are types."