Menu Merlin: From Summer AI Hackathon to Reality

From Hackathon to the Onboarding Floor

Last year Toast ran an AI-focused hackathon with intentional learning, collaborative building, and accelerating our work through AI in mind. Our recap of the event describes how we reimagined it from previous hackathons: extended to a full week, opened to every Toaster regardless of role or technical background, with in-office activities to bring back that classic, scrappy hackathon spirit. 

Menu Merlin - the hack that our team worked on and that we will talk through - was one of the fan favorites that came out of hackathon: an internal tool that takes a photo or a PDF of a restaurant's menu and produces a menu template our onboarding teams can work from. As we thought through our hack, we also thought through how we would keep the tool active and iterated upon it after the hackathon. Having an actual plan for how projects move forward afterwards was part of our plan well. But the week we built it is not the only part of the story. The interesting part is the months that followed, spent in close partnership with the onboarding organization, turning a proof of concept into something onboarding consultants now reach for as a matter of routine.

The Scrappy Spirit, Before It Was a Hackathon

That scrappy spirit is not new here, and Menu Merlin was not the first time this spirit showed up in the world of menu onboarding. Back in 2020, when restaurants needed to get online in a hurry and nobody had the luxury of a proper roadmap, Toasters built internal tooling that bridges a restaurant's existing menu into a Toast menu via a menu template that can be uploaded. It was put together quickly, under real pressure, to solve an urgent problem for restaurants under the economic pressure of COVID. 

Figure 1: Menu onboarding process

From a customer perspective, this menu template represents a significant hurdle simply to understand how to complete a template - an educational lift that becomes irrelevant after submission of the template. It still underpins menu onboarding: it is wired into the workflow our onboarding teams follow every day and a considerable amount of institutional knowledge has accumulated around it. 

With that context in mind, we developed Menu Merlin to address these two important realities. The first is a reminder worth stating plainly: scrappy work built under pressure can outlive the emergency that produced it by years, so it is worth building even when the circumstances require a less-than-perfect product at the first go. The second is more practical. That tooling set a target that we did not want to continue feeding into. We wanted to work with it with an eye towards making it a stepping stone to something better.

Everything else followed after that. By the end of the week we had a working web app, a service behind it, and an export that landed in the same menu template onboarding tooling we already had in place.

Figure 2: Menu Merlin proof of concept

The Technical Approach: Menus In, Templates Out

Architecturally, Merlin is not complicated, and we worked to keep it that way. A web app collects files, a backend service normalizes them into images, a commercially available multimodal LLM extracts structured menu data, a person reviews and corrects it, and the service writes the result into a menu template that onboarding already exchanges with restaurants for QA and for upload into the Toast system. 

Multimodal models, not an OCR pipeline. A traditional pipeline would leverage Optical Character Recognition (OCR) to recognize characters, reconstruct layout, then infer meaning, potentially compounding errors at every stage and introducing another step in the pipeline. Handing the image to a model that reads layout and semantics together collapses those stages into one, albeit with the usual trade-off dilemma in software engineering: we swap the convenience of having one stage while depending solely on the model. We run the risk of hallucinations, so we constrain the domain rather than trust it to do the job perfectly with a minimalist prompt. In the scope of the tool, we forced every extraction to be schema-constrained and we asked the model to report its own uncertainty in a commentary field rather than guess silently. 

Large menus forced us to be asynchronous. We started with a synchronous approach for the sake of speed, but we quickly ran into performance issues with uploading large menu files. The solution to that was making the process asynchronous so the user could upload the file and come back later once that backend finished processing the menu. 

Progressive fallback, because an editable draft beats an error. Models can fail: they rate-limit, they time out, token limits are exceeded, and they get deprecated on a vendor’s schedule (and not ours). Each model in our configuration names the next lower tier to try, terminating in an explicit "give up," and the app shows the user some output rather than doing nothing. A menu whose latest attempt errored now resumes from the last good version instead of becoming unreachable. 

Editing menus through natural language. Making edits to a menu was not initially accounted for in the most basic flow we started with: whole menu in; instruction applied to parse the image into a menu schema for the menu template; whole menu out, schema-validated. In working with our users, we discovered that it was important to give them the ability to allow them to edit as they reviewed the review. The process then became conversational in two steps: pairing the structured data with a natural-language response, then persisting conversation history so users could refer back to earlier changes. We additionally added in the ability of being able to go back and forwards in the conversation history to allow for maximum flexibility for onboarding in processing menus.

Prompt management as a production discipline. Our prompts live in a prompt management platform, not in the codebase, promoted by label rather than by deploy, with checked-in defaults as a fallback so an outage costs us prompt freshness and never availability. This gives us the flexibility to tweak prompts with onboarding subject matter experts without having to wait for builds to finish.

Figure 3: Menu Merlin after iteration

From Demo to Day-to-Day: A Partnership with Onboarding

Menu Merlin worked well in the demo, but getting it into the hands of real OCs, on real customer menus, under real deadline pressure, was a different project altogether. We shipped version one and then the harder work started. The question was no longer whether the model could parse a menu, it was whether the people building menus for customers every day would actually trust it enough to make it part of their process, and whether we'd hold up our end of the deal when they told us what was broken.

So we started small, and deliberately so. Rather than opening the tool broadly, we ran a pilot with a small cohort of expert onboarding consultants: people who had every reason to be skeptical but were willing to use something still in development for the sake of improving our customer journey, and who would tell us exactly where it broke or what wasn't working. We created dedicated Slack channels and structured feedback forms and we asked our users to submit feedback on whether the result was good or bad. The rating scale ran from "unusable, it would be faster to start from scratch" to "near perfect, ready to import with virtually no changes," and in the early months a single week routinely contained both. 

Users reported failures with a specificity we could act on directly. These were not abstract quality complaints: they were reproducible cases attached to real menus. Many were fixed the same day, most within a couple of days. By using a centralized prompt management solution, we were able to make fixes immediately without a code change.

The coordinators who'd been in the pilot from the beginning turned out to be the best advocates for the tool because they knew the workarounds, knew which menus to tackle first, and knew where to route a difficult file, so when their colleagues asked whether Merlin was actually worth using, they had a real answer. When we opened Merlin to the broader team we renamed the feedback channel rather than closing it, because the feedback was an asset. Onboarding leadership started building it into the standard workflow, the engineering team formalized the transition from proof-of-concept to production service, and the project started showing up in leadership calls as an example of what cross-functional AI work looks like at Toast.

Lessons Learned: Making a Hack Stick

Build with your users, not for them. The power user cohort built with us, told us the priority order of what to fix, surfaced failure modes we never would have found in internal testing, and explained in plain terms what the tool needed before they'd use it on a customer. This became a sustained, working relationship between the engineers who built Menu Merlin and the onboarding team who used it; both sides had to keep showing up for it to work. 

The human in the loop is a feature, not a limitation. There is no path through our app from upload to export that skips review, and before exporting, users acknowledge that Menu Merlin gives them a head start rather than a finished menu. We built human review into the process due to how we are leveraging imperfect emergent technology that, while improving every day, cannot replace a human eye for detail. Being direct about that is what made the tool trustworthy enough to expand.

Prompt management is a production discipline. Prompt engineering and management is a constant job with a tool like this. It allows us to tweak the experience without a full code change making the refinement quicker, smoother, with little overhead.

What's Next

We continue to refine the tool alongside the onboarding teams who use it, along with seeing where the tool may play a role in the broader Toast service ecosystem, with our office hours and feedback channels still open. The interesting work now is less about extraction accuracy and more about what the next generation of AI-assisted onboarding looks like when the tool participates in the workflow rather than sitting beside it. We are learning that in the same way we learned everything else here - with the people who do the work.

Acknowledgements

Thanks to Roberto De La Vina Estrada-Nora and Stacy Nguyen for being a part of the original Hackathon team; to Steve Watkins, David Young, Ruben Cagnie, and Dan Chu for code reviews and guidance during the hackathon; and to Martin Kressirer and Dan Teoh on the platform, legal, and AI governance side for their support. 

Most of all, thanks to the onboarding consultants participated in our pilot: Ann Simeoli, Ashley Bejsovec, Carrie Scott, Caoimhe Shanley, Emily Murphy, Kelsey Perry, Danielle Stack, Allison Ray, Bailey Miller, Sasha Gomez, Ryan Hartwell, Jerald Ortega, and many others.

____________________________

^{This content is for informational purposes only and not as a binding commitment. Please do not rely on this information in making any purchasing or investment decisions. The development, release and timing of any products, features or functionality remain at the sole discretion of Toast, and are subject to change. Toast assumes no obligation to update any forward-looking statements contained in this document as a result of new information, future events or otherwise. Because roadmap items can change at any time, make your purchasing decisions based on currently available goods, services, and technology.  Toast does not warrant the accuracy or completeness of any information, text, graphics, links, or other items contained within this content.  Toast does not guarantee you will achieve any specific results if you follow any advice herein. It may be advisable for you to consult with a professional such as a lawyer, accountant, or business advisor for advice specific to your situation.}