Documenting a project well is one of the most important aspects of the development lifecycle of any project as at the end of the day this will be shared and communicated with all the stakeholders, for example, clients, project managers, other developers etc whosoever associated with the project. Now descriptive and easy to understand images and diagrams are one of the most important features of any well-written document as they convey more than a thousand words. In this article, we’ll discuss one python package called Diagrams that will make it easy for you to create and maintain technical diagrams. Although this is a python package, it can be used for any project irrespective of the tech stack as we are never going to run the code on a production server.
Why python Diagrams?
Before delving deeper, we need to understand how this package exactly can help us and why would we want to use it in the first place over other traditional online and offline tools (e.g MS paint) etc. I feel below 5 main points makes the python Diagrams stand out from traditional tools.
For a quick and simple diagram traditional tool might just work fine but it becomes extremely difficult to describe a project with complex architecture/workflow and it becomes almost impossible to draw a diagram after a certain level of complexity. Python diagrams come here to rescue us as it enables us to create a diagram as code and like any well-written code, this also can be made modular, scalable and maintainable.
For a traditional drawing tool, it becomes a major challenge to maintain a diagram as the project become more and more complex and new component and interrelation amongst them get added to the project it becomes almost impossible to scale and maintain it. On the other hand, for python Diagrams, it’s pretty easy to extend the code as per need and it passes the test of time.
Python Diagrams offers much more flexibility. For example, if someone wants to focus on a small section of a large diagram. It’s easy to comment out part of the code and focus only on the required portion. For the same reason, it becomes much easier to try out different things and experiment with Diagrams.
For most of the traditional drawing tools, the options to depict a component are very limited which makes it difficult to follow as the diagram gets more complex. For the python Diagrams there huge collection to depict a component, cluster them for better understandability.
Code is reusable and version control is easier just like any other code.
Installation:
Diagram requires the system to have python 3.6 or higher. In case you don’t already have it you can download and install the required version from here.
Secondly, you need to install an open-source tool to draw graphs, which is used by the Diagrams package under the hood called Graphviz. This can be downloaded and installed from here. Just a word of caution that installing Graphviz using a python package manager like pip might not work.
Now install diagrams using any of the python package managers by running any of the below commands in the terminal.
# using pip (pip3)
$ pip install diagrams
# using pipenv
$ pipenv install diagrams
# using poetry
$ poetry add diagramsCheck if the diagrams got installed successfully by running
$ pip listCreate the First diagram
Let’s create our first diagram and understand the basics to use the package.
from diagrams import Diagram
from diagrams.aws.compute import EC2
from diagrams.aws.database import RDS
from diagrams.aws.network import ELB
from diagrams.aws.integration import SQS
from diagrams.aws.storage import S3
from diagrams.saas.chat import Slack
attributes = {"pad": "1.0", "fontsize": "25"}
with Diagram("message flow", show=True, direction="LR",
outformat="png",
graph_attr=attributes):
elb = ELB("load balancer")
service1 = EC2('Service1')
service2 = EC2('Service2')
service3 = EC2('service3')
db = RDS("primary DB")
elb >> service1 >> db
elb >> service2 >> db
service1 >> SQS('sqs') >> service3 >> S3('s3')Upon running the code above a diagram pop up as below and the same image is saved as mvc.png in the workspace. This diagram depicts a typical workflow in a microservice-based architecture. where multiple services work together, gets a request from the load balancer, access RDS dB, drop a message in the message queue and drop stores data in dB or storage service like s3, sending slack notification etc which are pretty self-explanatory.
A simple message flow in a microservice-based architecture
Now let's delve into the code and have a closer look. We first import the required class Diagram from the package diagrams. Then we import a few node classes from the AWS provider e.g. ELB, EC2 etc. You may explore all the node classes from different providers from the official documentation.
Now we create a new Diagram instance with the name “message flow”. As we’ve set property show = True, python would immediately open up the image as we run the program which helps in reducing debugging time. The third property direction implies the direction in which images start and grow. In this case, it is from the left to right direction (LR), which is the default. Other options are RL (right-left), TB(top to bottom), BT (bottom-top). Then we added some attributes of the diagram as dictionary and output image format as png. Now under the diagram, we create a few instances of nodes that we already imported. Now to create a directed edge between any two nodes all we have to do is to join them using “>>” or right shift operator if you say so.
Adding More descriptive Edges
In the above example, we used the “>>” operator which added a default edge. But in a lot of scenarios, it would make a lot of sense if we can make the edges more descriptive, customise style it as per our requirement. This can be done by using a class Edge. Let’s suppose we make the below modifications in our code and we want to highlight the edge towards slack notification.
from diagrams import Edge
service3 >> Edge(label='sending error notification', color='red', style='dashed') >> Slack('slack notification')
Adding custom edge
First, we had to import the Edge class. It takes arguments like label, colour and style to customise the edge and then add an instance of it to the diagram just as any other node.
Adding custom node
Now, what happens if you want to send SMS and email notifications as well besides slack notification? And guess what, the diagram package does not have a class for them. In such a scenario, or if you are a logo designer and you want to use your own custom logo for node instead of the default one, you can use Custom class to use your own logo. Now we can easily create our own custom email and SMS notification node and add them to the diagram as below:
from diagrams.custom import Custom
email = Custom('Email notification', 'email.png')
sms = Custom('SMS notification', 'sms.png')
service3 >> Edge(label='sending error notification', color='red', style='dashed')
>> [Slack('slack notification'), email, sms]
Adding custom nodes
Now as expected we have successfully added our custom
email and SMS notification to the diagram.
Grouping and clustering of nodes
Now, in the last example, you might have noticed that there are multiple service nodes or EC2 instances are connected to the elastic load balancer. What would happen if the number of service nodes increases? We would end up creating a lot more identical ages from ELB to those nodes making the code messy and harder to maintain. But don’t worry. We can simply group the nodes in a list and treat that as a group. For example, in the previous code snippet, instead of treating two service nodes as a separate entity, we can put them in a list and treat them as the same entity while adding ages. e.g.
service1 = EC2('Service1')
service2 = EC2('Service2')
service_node_group = [service1, service2]
elb >> service_node_group >> dbThis will create the same diagram as above while making the code easier to understand and maintainable.
Now, in a complex architecture, nodes can be segregated into different logical entities called clusters to make the diagram even more modularised and easy to understand. For example, in our example, we can cluster all related microservices, databases, static data stores, notifications into separate clusters. To give an example, code for the cluster for all the services would look like
from diagrams import Cluster
with Cluster("Services") as services:
services = [EC2("Service1"), EC2("Service2")]Incorporating all the concepts we discussed so far our final code looks like
from diagrams import Diagram, Edge, Cluster
from diagrams.aws.compute import EC2
from diagrams.aws.database import RDS
from diagrams.generic.database import SQL
from diagrams.aws.network import ELB
from diagrams.aws.integration import SQS
from diagrams.aws.storage import S3
from diagrams.saas.chat import Slack
from diagrams.custom import Custom
attributes= {"pad": "1.0", "fontsize": "25"}
with Diagram("Sample data flow in microdervice architecture",
show=True,
direction="LR",
outformat="png",
graph_attr=attributes):
with Cluster("Service layer") as services:
service1=EC2('Service1')
service2=EC2('Service2')
service3=EC2('service3')
request_handlers= [service1, service2, service3]
with Cluster("Elastic load balancers") as elbs:
elb1=ELB("Load balancer1")
elb2=ELB("Load balancer2")
elbs= [elb1, elb2]
with Cluster("Database layer") as dblayer:
rds_dbs= [RDS('db1'), RDS('db2')]
sql_db=SQL('db3')
with Cluster('Batch processing layer') as batch_processors:
service4=EC2('Service4')
service5=EC2('Service5')
batch_processors= [service4, service5]
with Cluster('Notification channels') as notifications:
email=Custom('Email notification', 'email.png')
sms=Custom('SMS notification', 'sms.png')
notifications= [Slack('slack notification'), email, sms]
elb1>>Edge(label='requests') >>request_handlers
elb2>>Edge(label='requests') >>request_handlers
service1>>Edge(label='store data') >>rds_dbs
[service2, elb1 >> Edge(label='requests') >> request_handlers
elb2 >> Edge(label='requests') >> request_handlers
service1 >> Edge(label='store data') >> rds_dbs
[service2, service3] >> Edge(label='store data') >> sql_db request_handlers >> SQS('sqs') >> batch_processors
service4 >> Edge(label='sending error notification', color='red', style='dashed') >> notifications
service5 >> Edge(label='sending error notification', color='red', style='dashed') >> notificationswhich yields the below diagram:
Feel free to play with the code to generate even better and expressive diagrams.
limitations
Edge can be there only from one node to another i.e. we can’t add any edge at the cluster level.
Edge from one list of nodes to another list of nodes is not supported.
Conclusion
In this article, we discussed why we should python package to draw technical diagrams. This code can be part of the repo itself as a doc and can be updated from time to time and the generated diagram can be used in all kinds of technical presentations e.g. read me, ppt or confluence pages etc.
Source: Medium - Arnab Sen
The Tech Platform