database-sharding

Materialized view is the cache of views. It pre-computes, stores, and optimizes data access when created, and automatically refreshes to ensure real-time data availability.

Materialized view pattern

Multi-column index in DB

A comprehensive guide to understanding circular references in database design, including types, challenges, real-world applications, and solutions for managing them effectively. Covers self-references, circular dependencies, and strategies for maintaining data integrity while working with circular relationships.

Database design circular

When we started working on this project aimed at the creative community, we faced an interesting challenge. We needed to build a model that was similar to a task but with a key difference: it had to support custom, dynamic, and extensible properties. If you’ve ever used Notion and appreciated how flexible its objects are, you’ll know exactly what we were trying to achieve.

Designing a model with dynamic properties

As systems scale to meet growing demands, databases must evolve alongside them to maintain performance and integrity. This document explores best practices for managing database changes, maintaining knowledge, and ensuring smooth integration. Topics covered include knowledge sharing, repository structuring, continuous integration, and database refactoring, with real-world examples illustrating their application.

Evolutionary database design: managing change and scaling with the system

The motivation of this thoughts that trying to discover the LLMs and Generative AI...

Quick learning vector database

Learn how to choose between raw SQL, ORM, and SQL generators, optimize database performance with smart indexing, avoid rounding errors, and write efficient queries for better security and speed.

Sql practices orm vs plain sql

Structured Query Language (SQL) is widely used for managing data in relational database management systems (RDBMS). In the context of SQL, the optimization of database queries forms the foundation for efficient data retrieval, providing quicker response times and increasing overall database performance. One key strategy to optimize SQL queries involves the concept of "Sargable" queries...

SQL saragable queries and their impact on database performance

This article explores chunking strategies to handle context limitations in Large Language Models (LLMs) like GPT. It covers preprocessing data, selecting chunk sizes, and how to maintain coherence in various content types such as unstructured, Markdown, and LaTeX.

Chunking strategies to overcome context limitation in LLM

An overview of how to implement long-term memory in a ChatGPT-based chatbot using a Vector Database (VectorDB) to store conversation history and retrieve the most relevant past interactions.

Storing long-term memory in ChatGPT using VectorDB

An exploration of query caching in Large Language Models (LLMs), focusing on how semantic vector databases can drastically improve efficiency and reduce computational costs by reusing cached answers for semantically similar queries.

Query caching for large language models

In spite of having taken the world by storm, Large Language Models(LLM) still has some limitations such as limited context window and a knowledge cutoff date. Retrieval-Augmented Generation(RAG) steps in to bridge this gap by allowing LLMs to access and utilize external knowledge sources beyond their training data. However, data is not text based only, it also can be image, audio, table in docs,...

Multimodal: in rag

An overview of vector databases and their application in LLM systems. This article covers the history and core concepts of vector databases, their use cases, implementation considerations with LLMs, and a comparison of popular vector database options.

Select vector database for LLM

An exploration of natural language to database query systems using MongoDB, examining system prompts and implications for data engineering and agentic workflows.

Natural language to database queries: Text-to-MongoDB

A migration-less, append-only database design using TimescaleDB's hypertables and continuous aggregates to capture observational data and detect emerging patterns through LLM analysis.

Database design and philosophy

Discover how a trading platform mitigated database access risks, enhanced security, and ensured data integrity through role-based access control, network isolation, MFA, and robust logging. Learn about the strategies and tools, like Teleport, that transformed operational efficiency and reinforced client trust.

#database

C

D

E

M

N

Q

S