Suppose an API's p95 latency rises while database CPU remains moderate. One possibility is that requests wait for a pooled connection before they send any SQL. Query duration alone will not reveal that queue.
1. Start the request span
Record a request start time and trace identifier. Add spans or metrics around connection acquisition, query execution, and response serialization. Avoid placing secrets or personal data in trace attributes.
2. Acquire a connection
The request asks the pool for a connection. If one is idle, it can proceed. If all are busy, it waits in a queue subject to pool and request timeouts. Rising concurrency, long transactions, leaked connections, or a pool sized too small for the workload can increase wait time.
3. Execute and release
Once acquired, the connection sends a query and waits for the database. After the result is consumed and transaction state is handled, the application returns the connection to the pool. Holding it while doing unrelated network calls reduces capacity for other requests.
4. Compare the timings
Break end-to-end latency into acquisition wait, query time, downstream work, and response time. Check histograms, queue depth, active and idle connections, timeouts, and database wait events. Increasing pool size without checking database capacity can move the queue into the database and make the system less stable.
The trace turns “the database is slow” into a testable hypothesis about where the request waited.