A Story of Failure; BASE Tech: Full Text Search Engines
How an engineer who screwed up twice didn't get fired, and why you should use a full-text search engine instead of your database.
Welcome new subscribers! I had a small win sharing the newsletter on LinkedIn this week. It’s a pleasure to write for you and elevate your engineering careers.
Today for you I have:
A Story of Failure
BASE Tech: Full Text Search Engines
Links & Happenings
A Story of Failure
An upper mid-level engineer on our team deployed consecutive releases to production that caused bugs or failures.
You can probably relate to how they felt. Terrible at their job. A disappointment.
This dev was certain they were going to be fired.
But we had a positive culture in our engineering team. We used our regression protocols to revert the deployments. We let them immediately fix the issues and deploy the corrected features to production. We analyzed how the bugs eluded our review and testing processes so we could better protect against future occurrences.
All without blame.
Because it happens to us all. Releasing a bug to production is a right of passage. (Hopefully one that doesn’t have too big an impact on the business!)
This dev knew they made a mistake, was empowered to fix it, and learned from the process. They improved their experience and skills, and gained trust in the organization that didn’t punish them.
Senior engineers acknowledge the fallibility of humans, introduce and encourage a culture of support, ensure technology is in place to help protect against mistakes, and encourage autonomy.
And we learn from our mistakes. 🙂
Do you have a similar story about yourself or a colleague? Leave a comment or hit reply and tell me about it!
BASE Tech: Full Text Search Engines
💡 Each BASE Tech briefly covers a technology, its purpose, and how a Senior engineer might approach using it.
You want to add search to your app, and all your data is currently stored in a DB such as MySQL. The first solution you might turn to is LIKE ‘%query%’ SQL queries. No new software and it “gets the job done”, right?
Searching is complex, and not just in implementation. A great search experience includes many features and nuances that are not provided by LIKE SQL queries:
Language and Text Analysis – Fuzzy matching common typos, stemming (reducing words to their base or root form), lemmatization (considering the context to convert a word to its meaningful base form), and handling synonyms. Such features improve the search's ability to match on the intended meaning rather than just the exact text.
Relevance Ranking – Rank results based on their relevance to the search query. The most relevant results are returned first, improving the user experience. Many engines also provide highlighted relevant terms you can use in your UI.
Advanced Search Features – Boolean operators, phrase matching, proximity searches, wildcards, and more. These features allow users to perform more complex and precise searches.
Performance – Full-text search engines are optimized for searching text data efficiently. They use specialized indexing strategies that allow for much faster search operations, especially across large datasets.
LIKEqueries, especially those with leading wildcards, can be very slow because they often require scanning entire tables or columns to find matches.Customization – Tailored indexing, field-level configuration, custom ranking algorithms, and more.
A full-text search engine empowers you to provide the search experience your users expect (Google and major tech has spoiled them all). How does a Senior make that happen?
Gather requirements, choose an engine, and create an implementation plan that includes development, pre- and post-launch phases:
Gather Requirements – Understand your users’ needs of what they are searching and why. Use this to define the scope of searchable content – the fields, data, and types to be searched. Don’t forget requirements like security and compliance.
Choose an Engine – Find one that has the features you need based on your requirements, while balancing performance, scalability, cost, and ease of integration.
Model and Index Your Data – Design a schema that represents your searchable content. Develop the process for indexing your content, ideally an automated one to ease deployment and ensure consistency.
Build the Thing! – Integrate search into your backend, creating service/repository/data layers and abstractions, calling APIs, and anything needed for indexing and querying data. Implement mechanisms to keep the search index up to date with data changes.
UI/UX – Design and implement the search interface. Add delighting features like autocomplete, faceted search, filters, and sorting where applicable.
Test and Optimize – Stress test the search implementation under various loads. Adjust ranking algorithms and relevance settings to improve the quality of results based on user feedback and analytics (have a plan for this, it isn’t an afterthought!)
Monitor and Maintain – Track usage, performance, and health of your search infrastructure. Understand what maintenance you’ll need over time, such as software updates and scaling as your data volume grows.
You’ve now planned for and implemented a powerful and useful full-text search feature just like a Senior engineer would! 👏🏼
Here are some engines I recommend looking into for your search needs:
There are more open source and paid hosted solutions out there, if none of these suit your needs. I’ve used Solr (and Sphinx, which doesn’t seem as modern or maintained as others) in past projects. More recently I implemented Meilisearch for SpendBetter.com, then switched to Algolia when I needed multiple geo coordinates per record.
Have you implemented full-text search? Use a different engine than one mentioned here? Let me know how your experiences went!
Links & Happenings
I recently upgraded my laptop, and with a fresh start have been trying new software: Warp, a modern, Rust-based terminal, and GitKraken, a git GUI. Yeah, I’m a GUI guy, let’s fight about it!
My friend Andy is experimenting with Airplane, a developer platform for internal tools. It has a self-hosted option which is unique for the space.
I’m creating a silly API project this month and Zuplo is on my radar as an API Gateway layer. They also have a silly-but-useful tool: RateMyOpenAPI. Shout out to Adrian @ Zuplo for sharing!
Last night I tried the Sleep Cycle app for the first time in an effort to improve my morning routine. It worked pretty well as an alarm and captures some interesting data (and sound clips of my wife snoring 😴). I’ll be curious to see if results stay consistent. Shout out to Zach Highley’s How To Become a Morning Person YouTube video for the recommendation.
Happy Wednesday! Come chat with me on LinkedIn, Twitter, or hit reply and let me know how I can help with your engineering career.
Aken 💙