How idempotency helps you in building safer and more scalable software systems

An important software concept for building better systems.

Published in

Dev Genius

7 min readDec 11, 2023

Shopify Engineering has a fantastic blog post called 10 Tips for Building Resilient Payment Systems. One of their most interesting insights comes from point 6- Use Idempotency Keys, where they have the following to say-

Distributed systems use unreliable networks, even if the networks look reliable most of the time. At Shopify’s scale, a once in a million chance of something unreliable occurring during payment processing means it’s happening many times a day…
Our centralized payment service can track attempts, which consists of at least one or more (retried) identical API requests, by sending an idempotency key that’s unique for each one…If any of these steps fail and a retried request with the same idempotency key is received, recovery steps are run to recreate the same state before continuing.
… In one high-throughput system at Shopify we’ve seen a 50 percent decrease in INSERT statement duration by switching from UUIDv4 to ULID for idempotency keys.

This made me think of how overlooked idempotency can be. My computer science degree had multiple classes in software engineering, and none mentioned this idea. I’m not going to let our cult make the same mistake, so here is an article on idempotency.

Source

Join 150K+ tech leaders and get insights on the most important ideas in AI straight to your inbox through my free newsletter- AI Made Simple

Key Highlights

What is idempotency?- An operation that produces the same output regardless of the number of times it’s executed, with the same input. This Ensures data consistency and prevents duplicate operations.
Why is idempotency important?- Idempotency gives us Resilience, Scalability, Performance, Simplicity, and Provability.
How is idempotency implemented?- The best way to implement idempotency is to utilize good design. Some common techniques include Unique identifiers (this is what Shopify did), Idempotent API design, and versioning. We’ll cover some case studies.
Challenges of idempotency: As is the motif with any idea that improves performance and scale in distributed systems, proper planning is key. The only guarantee with idempotency is additional overhead. The gains are worth it, but you have to plan it well.

Let’s cover these ideas in more detail.

What is idempotency?

In Software terms, Idempotency refers to the property of an operation that ensures its repeated execution, with the same input, always yields the same output regardless of the number of times it’s executed. In other words, a subsequent execution of an idempotent operation has no further effect on the system’s state. This characteristic plays a crucial role in building reliable and robust software, especially in distributed systems and fault-tolerant environments.

This is taken from Math. Idempotent functions in Math refer to functions where (f(x))^n= f(x) where (f(x))^n refers to applying a function f to itself n times (n being a positive int). (f(x))² would be f(f(x)). Functions like deleting an element from a set are idempotent because you can spam those deletes w/o worrying about a crash.

You might not see the uses for idempotency here. Let’s go into them.

Importance of Idempotency:

Idempotency gives our systems the following benefits-

Resilience: Idempotency helps software handle network failures and retries gracefully. If a network request fails, the client can resend the request without worrying about unexpected side effects. This ensures data consistency and prevents duplicate operations.
Scalability: In distributed systems, tasks are often distributed across different servers. Idempotency allows multiple servers to process the same request without affecting the overall outcome, improving scalability and fault tolerance.
Performance: By guaranteeing the final state after an operation, idempotency can optimize caching mechanisms. Servers can cache the results of idempotent operations, leading to faster response times.
Simplicity: Designing idempotent operations simplifies code and reduces the complexity of error handling. Developers can focus on the core functionality without worrying about side effects caused by repeated executions.
Provability: Only nerds care about this, but idempotency allows you to build provably safe systems. Since you don’t have side effects, you can predict how your systems will behave (at least theoretically). Think back to Functional Programming and pure functions; it’s a similar principle.

Sound useful? Let’s cover some details.

Implementation Strategies:

You can implement idempotency with the following methods-

Unique identifiers for operations: Assigning unique identifiers to each operation allows the system to track its execution and prevent duplicates. These identifiers can be generated by the client or the server. Shopify did this by using timestamps in the keys, making them unique (and lexicographically sortable). “We prefer using an Universally Unique Lexicographically Sortable Identifier (ULID) for these idempotency keys instead of a random version 4 UUID. ULIDs contain a 48-bit timestamp followed by 80 bits of random data. The timestamp allows ULIDs to be sorted, which works much better with the b-tree data structure databases use for indexing.” It also is a good example of a principle I try to highlight in both my publications: good design is the best safety technique. In this case, a good data model saved Shopify a lot of headache.
Versioning: In a similar vein, keeping track of the operation’s version helps ensure that only the latest version is applied. This is particularly useful for long-running operations that may require multiple steps. One way to do so is to store the state of your program as the version code, using the states to catch repeated operations.
Transaction management: Utilizing database transactions allows for the grouping of multiple operations into a single unit. If any operation within the transaction fails, the entire transaction is rolled back, maintaining data consistency. Make sure you don’t group too much together, because reversions can then lead to a bad user experience.
Leveraging HTTP Methods for Idempotency: For microservices, “maximize the inherent idempotent properties of HTTP methods to design interactions that align with idempotency principles. Capitalize on the reliable behavior of methods like PUT and DELETE to craft consistent operations regardless of retries or failures”.

Case Studies:

Some common use cases for idempotency include-

Payment processing: In payment systems, charging a credit card multiple times for the same transaction must be avoided. Implementing idempotency ensures that only one charge is processed, regardless of how many times the payment request is sent. An idempotent payment process can be seen below-

Source

Database management: Some database management operations are idempotent. Amazon’s DynamoDB is a good example of how grouping and database management can leverage idempotency.

HTTP- In web development, HTTP methods like GET, PUT, and DELETE are designed to be idempotent.

As mentioned, the additional memory overhead in storing additional information to track the version can be a lot. Designing with that in mind is key. But when done right, idempotency is a huge addition to your systems.

Do you have any experiences you’d like to share? You know how to reach me.

That is it for this piece. I appreciate your time. As always, if you’re interested in working with me or checking out my other work, my links will be at the end of this email/post. If you found value in this write-up, I would appreciate you sharing it with more people. It is word-of-mouth referrals like yours that help me grow.

Save the time, energy, and money you would burn by going through all those videos, courses, products, and ‘coaches’ and easily find all your needs met in one place at ‘Tech Made Simple’! Stay ahead of the curve in AI, software engineering, and the tech industry with expert insights, tips, and resources. 20% off for new subscribers by clicking this link. Subscribe now and simplify your tech journey!

Using this discount will drop the prices-

800 INR (10 USD) → 640 INR (8 USD) per Month

8000 INR (100 USD) → 6400INR (80 USD) per year (533 INR /month)

Get 20% off for 1 year