What Ticketmaster is doing about technical debt

http://tech.ticketmaster.com/2015/06/30/what-ticketmaster-is-doing-about-technical-debt/

by kilimchoi • 11 years ago

38 51 11 years ago

tech.ticketmaster.com

nanis• 11 years ago

I never had a good experience using Ticketmaster. I doubt more tests is going to solve that.

degenerate• 11 years ago

Look at them cleverly placing themselves next to prominent systems like the air traffic controller, while toting their millions of lines like it's some sort of accomplishment. For a business that really only needs to accomplish two things to make their customers happy: 1) print tickets and 2) stop scalpers, they do absolutely terrible at those 2 things. I don't mean terrible as in they do the job poorly, it's just that they care 0% about the user experience during the entire ticketing process from end-to-end.

smackfu3• 11 years ago

Ticketmaster is essentially enterprise software-as-a-service: the people buying it (the venues) are not the people using it. From that perspective, complaints are not surprising at all.

slg2• 11 years ago

Except a large part of the Ticketmaster system is back of the office stuff that the consumer never uses. It isn't only the system used for the end user to purchase tickets, but also the system that the venue uses for things like inventory and access management. Odds are if you ask people working at the venue, they will have just as many complaints about Ticketmaster as you do. The problem is that none of their big competitors (that I have seen at least) have really put together a product as complete as one that Ticketmaster can offer a venue.

walshemj• 11 years ago

Agreed I helped out a uk cinema system with their system based on the effective monopoly supplier's system.

It was "interesting" in the Chinese sense but an absolute PITA to work with.

And that's not mention the WAN linking the cinemas based on packet radio that blocked :-)

olalonde• 11 years ago

They used to have their own operating systen or something according to an article I read about Terry Davis though that time is probably long gone. Edit: found the article: http://motherboard.vice.com/read/gods-lonely-programmer

SEJeff• 11 years ago

So Denny Denker, the Ticketmaster chief architect, wrote a VAX Assembly language realtime ticketing operating system. He wrote it in the 70s, before things like TCP/IP were standardized, and they managed to virtualize it on Linux using a hacked up vax emulator[1]. They've tried many many times (when I was there around 2007ish) to rewrite it using normal things like C++ and mysql, yet couldn't manage to get it faster than a specialized realtime operating system. It was kind of hilarious when we moved it to NFS, as the operating system timed writes to the speed of the PDP-11 drive spin rate. When we moved it to NFS, the IO was 1000x faster, and the entire thing crashed. No end of jokes resulted from these sorts of "quirks" in our systems.

Last I've heard, they've been slowly moving functionality out of the VAX operating system into a more traditional stack.

Source: I used to work on the team responsible for the Linux servers that ran these virtualized vaxen, check the resume linked in my profile.

[1] http://simh.trailing-edge.com/

funkyy• 11 years ago

I have to agree to that.

Once I have bought tickets, traveled 500 kilometers, stayed at hotel near venue. Once I got there, there was dozens of people confused about place being shut. It turned out that the gig as cancelled and Ticketmaster forgot to notify buyers about it. This was for one of the well known US bands, so there was a lot of people there. The reason of cancellation was sickness of one of the members of the band that was announced on other site also selling tickets a week earlier so only Ticketmaster customers were affected.

Imagine disappointment. Also, I didn't get refund for tickets either despite requesting it.

cdcarter2• 11 years ago

Actually, to make their customers happy, they need to provide high quality inventory and revenue management tools that tightly integrate with third party databases and systems, and allow their users to make strong business and pricing decisions based on their data.

Ticketing systems are products sold to a presenter or promoter (or venue) and facilitate their business.

kbenson2• 11 years ago

For a business that really only needs to accomplish two things to make their customers happy: 1) print tickets and 2) stop scalpers, they do absolutely terrible at those 2 things.

Their job isn't really to do either of those things, and I'm not being glib. Let me explain:

First, TicketMaster (TM) doesn't really want to print your tickets. They would much rather just digitally manage them, and let you use your phone to go the the venue. Non-digital media costs real money, and TM would prefer to charge you for the convenience of managing your tickets, or sending you a PDF. As a bonus to them, this lets you easily resell your tickets as a fan on their TM+ exchange.

Secondly, TM doesn't really want to stop scalpers, at least not completely. This one is a bit complex, because portions of TM are definitely not scalper (i.e. broker) friendly, while other portions are. TM runs their own secondary market, called TM+, which is a "fan resale" market, in which they list ticket that were sold on the primary market (i.e. TM) for resale, with the benefit that since they were the original seller, they actually reissue the tickets with new barcodes and they are guaranteed to be accurate and work for entry. They are quite happy to work with the broker industry on this side, and will integrate with larger brokers to allow listings to be easily updated.

Personally, I think the bad names brokers get as a whole are often unwarranted, as there is a lot of benefit to a secondary market that people often ignore when they see extremely expensive tickets, such as extremely underpriced tickets for events that bomb (or another date is added), and the ability to find tickets for an event long after it's sold out (without a resale market, you would have little recourse to see an event that sold out minutes after it went on sale unless you got your tickets right then). That said, there are some bad actors in this arena, which brings us to the following...

I don't mean terrible as in they do the job poorly, it's just that they care 0% about the user experience during the entire ticketing process from end-to-end.

TM has a really hard job. I'm not sure you understand the scale of traffic they get on a popular event's main sale. They can literally sell out a 30,000 seat venue in seconds. Keep in mind, on a sale that's busy like that, there are often brokers (the bad actors referenced before) that are running industry software that leverages AWS/GCE/Linode/Rackspace, etc to access the main sale page tens to hundreds of times a second per instance during that sale to get CAPTCHA words and answer them. Since the CAPTCHA success rate is relatively low, they brute force it with many, many parallel requests. Consider that during peak sale season, there may be multiple of these large sales per hour, all set to open on the hour. TM is constantly battling these brokers by changing subtly how their ticketing system functions. It's not that they don't care about customer experience (although they do have enough of the market to not have to worry too much), but that it's only part of an equation that must also be balanced with insane levels of load balancing and queuing to handle the crazy load they see.

Now, I don't want to give the impression that TM is without fault. They've definitely got their fingers in multiple industries here which historically have had quite a bit of animosity, and they are fairly good at playing the media and the industries against one each other while secretly assuaging them in the background. The fact of the matter is that you as a customer aren't their only concern and aren't their only business. They are actively balancing the needs of the promoters, artists, venue and brokers, because their business caters to all of them.

Edit: Clarified scope of virtual server usage slightly.

KingMob• 11 years ago

Not to disparage the hard work of the techies there, but as a de facto monopoly, Ticketmaster can probably survive technical debt better than most.

astrodust• 11 years ago

They could just add yet another surcharge to cover it and what can we do?

Basically nothing.

joezydeco• 11 years ago

"Server operation fee....$15.00"

ape4• 11 years ago

Could somebody please disrupt this business! No user likes Ticketmaster and their "service fees".

avolcano• 11 years ago

Ticketfly is doing a good job. They do charge some fees, but not to the egregious extent of Ticketmaster, and have a much better interface that holds up against traffic.

Unfortunately, Ticketmaster has a habit of just buying/merging with competitors (including Live Nation, in a merger that should have never been allowed to go through).

Plus, Live Nation actually owns a lot of venues, so you'll never see them use another ticketing provider: https://en.wikipedia.org/wiki/Live_Nation_Entertainment#Venu...

imissmyjuno• 11 years ago

Also TicketWeb, and Northern Tickets (acquired by Ticketfly now?)

jasode• 11 years ago

You can't disrupt Ticketmaster's service fees because it's the venues and the peformers that are also profiting from those service fees.

Apparently, Ticketmaster's business model is based on taking the negative public relations hit for charging higher prices on behalf of their customers.

If you create a startup called CheaperTickets.com with the noble intention of not charging those hated service fees, you won't get any customers. (Keep in mind it's the venues & the artists that are the real customers of Ticketmaster and not the ticket buyers). The venues want the lucrative cut of the service fees while Ticketmaster gladly takes the heat.

[1]http://www.laweekly.com/music/ticketmaster-and-servants-band...

[2]http://www.reddit.com/r/todayilearned/comments/1xlt0i/til_ti...

ta92929• 11 years ago

That sounds plausible to me, I'm not an expert, but it's not always a choice for the bands.

Pearl Jam tried to stand up to Ticketmaster with a boycott and lost.

jakejake• 11 years ago

I don't know TM's numbers, but I know it is correct that processing fees are not always going 100% to the ticketing company. My company is in the music business and one of our products is for internal ticket sales. We collect fees on behalf of the seller. The sellers portion varies, but is usually about 75% of the fee - meaning that we, the processor, are only getting 25% of the fee that the customer sees.

jarmitage• 11 years ago

Enter Dice https://dice.fm/

sosuke• 11 years ago

They did, StubHub (not the only offender) buys bulk tickets, doubles the rates for you, and still adds service fees.

Ticketmaster is practically a saint next to them in my eyes.

chongli• 11 years ago

This just tells me that the original venues were not charging enough for tickets. If the tickets were priced ideally then StubHub would not be able to do this; the margins would be too low for the amount of risk involved.

ceejayoz• 11 years ago

Ticketmaster is practically a saint next to them in my eyes.

Ticketmaster owns a Stubhub competitor. https://en.wikipedia.org/wiki/TicketsNow

JoblessWonder• 11 years ago

Do you have proof of StubHub actually doing that? I buy/sell a lot on the secondary ticket market and don't think StubHub is purchasing tickets themselves (except to cover a gurantee.)

Now individual/corporate ticket brokers are for sure buying up bulk tickets and putting them on the secondary market through StubHub... but StubHub doesn't have anything to do with that besides being the platform utilized.

pizza234• 11 years ago

This business doesn't need disruption in the technical sense; there are plenty of alternatives, which in some cases, are even very easy to switch to.

The barriers are the very typical ones that big companies (as a matter of fact, I assume that the venue with the greatest exposure are in fact big companies) face towards innovation: bureaucracy, lock-in, inertia, PHBs, and so on.

Not that TM doesn't have any positive side - it's just a very typical example of the business workings of large companies (not all, of course).

nickpsecurity2• 11 years ago

Seems nobody has reviewed the actual article... So, I will: it's great. I could see this model, esp illustrations, getting somewhere with lay management that run things by the numbers and ROI. I'd like to try it on bankers given they're experts on getting people stuck in debt. Last thing they want to do is be caught in their own type of trap. I'd like to see more work like this, esp improving manual measurement areas.

goorpyguy• 11 years ago

I'll agree with you there, I like seeing the thought process that fed into this. We've had similar experiences having difficulty explaining why certain things are painful and then quantifying "how painful" and why. This seems to really try to address that, at least as a first step.

lectrick• 11 years ago

Can you summarize how they quantitatively measure aspects of tech debt? Because that seems to be a hard problem and I kind of don't believe that they have pulled off this feat

nickpsecurity2• 11 years ago

It's described on the page along with qualitative measurements they use. They used each where it made sense to produce something pretty solid for management to work with. The alternatives are currently... I don't know any...

So, until I see them, I think they did a good job going from using a vague concept to practical metrics. Baby steps, for sure, but with immediate, payoff potential.

dreamfactory2• 11 years ago

not that hard, they use http://www.howtomeasureanything.com/

JTon• 11 years ago

Firstly I'll admit, I don't buy many tickets in general. However the times I used Ticketmaster I've had a neutral to positive experience. I'm always a bit annoyed at the service fees. But it takes little time to buy them, and the tickets go straight into my inbox. Whatever. I think there are more pressing issues to complain about

PirateDave• 11 years ago

If you've ever tried to purchase tickets for a concert that you know would be sold out, the process is horrendous.

I mean, they know when they'll have a surge in customers (there's a freakin countdown clock to the purchase time) and yet they still can't set up something more elastic that actually allows me to load the page?

programminggeek• 11 years ago

They couldn't easily define or decide what technical debt was? Wow.

antrover• 11 years ago

What is Ticketmaster doing about outrageous fees?

knodi123• 11 years ago

chuckling nervously?

s1lver• 11 years ago

What about moral debt?

run4yourlives2• 11 years ago

Technical Debt Fee: $15.00.

Problem solved, no?

subpopular• 11 years ago

How an individual business has more lines of code than the largest and most complex scientific experiment ever conceived by human beings is...I dunno, embarrassing?

surferbayarea• 11 years ago

Strange how no-one here seems to have actually read the article. Do people have thoughts on how they handle technical debt and manage code quality? This was a well thought out way of structurally improving code quality in a code base that supports a big business driving 10billion$+ transaction volume. Do people have a better solution to what they should be doing?

RE:disrupting their business model, why is charging money for a service wrong? You cannot keep things for free by burning VC money. That only lasts so long...

What Ticketmaster is doing about technical debt

Related Stories