Welcome to the last part of our tutorial on building your own Twitter Analytics dashboard with Python, Azure Functions and Anvil.
Let us summarise what we have done so far.
- In the first part of our series, we got to know the Twitter API and learned how to retrieve Tweet stats and Follower/Unfollower data.
- In the second part, we used Azure Functions to periodically download and store these data to Azure Blobs
- In the previous blog post, we used Azure Blog Storage triggers to compare our point-in-time snapshots to their predecessors automatically. Thus, we compiled an aggregated result of our Twitter Followers over time and a consolidated data structure for our Tweet stats.
There is little value in any statistic unless it can be made visible. We still need a way to plot graphs of our Twitter performance over time. That’s why we will build a simple dashboard today. To make this article short and easy to follow, we will just focus on new and lost follower data over time. However, you will see how easy adding more graphs and insights into this dashboard will be.
To make a modern dashboard, we will build a web-based application. This means that you will be able to access your dashboard from within your browser. You have seen these kinds of applications quite frequently. Twitter, Gmail, Google Analytics,… – all of those run in your browser interactively.
You have two options to start programming with Anvil:
- You can download the open-source version and run everything on your own machine, or
- You can sign up for an Anvil account and use their service for your development.
I have outlined both variants in the Getting Started with Anvil Guide
Before we start, we create a skeletton app with Anvil. To do this, we create a blank Anvil app to start with
create-anvil-app hello-world MyTwitterDashboard
This will create a new Anvil app in a directory called
MyTwitterDashboard. Of course, you can name it whatever you want.
We can check whether the creation of the app has been successful and whether anvil starts up correctly by using the
anvil-app-server --app MyTwitterDashboard
If you now head over to you browser at localhost:3030 and see an empty page, everything worked fine.
Now, we create a server side function in anvil which is responsible for connecting to Azure and download our data. Head over to the
ServerModule1.py file in the
server_code directory. Here we can re-use the
get_blob_service_client function which we coded in the previous part of this series and expose two functions,
get_follower_data to Anvil’s server.
@anvil.server.callable will instruct Anvil to expose this function over HTTP, so that we can call it from the front end.
Note that we rely on Anvil’s secret storage. In this storage we place a variable called
AZURE_CONNECTION_STRING to access our Azure Connection String. This way you don’t have to hard-code your sensitive information into the code.
Here is the full source code of
import json import os import gzip import anvil.server import anvil.secrets from azure.storage.blob import BlobServiceClient def get_blob_service_client(): AZURE_CONNECTION_STRING = anvil.secrets.get_secret("AZURE_CONNECTION_STRING") blob_service_client = BlobServiceClient.from_connection_string(AZURE_CONNECTION_STRING) return blob_service_client def get_data_from_azure(blob_service_client, blob_name, container_name="output", default=None): try: blob_client = blob_service_client.get_blob_client(container=container_name, blob=blob_name) data = blob_client.download_blob().readall() uncompressed = gzip.decompress(data) return json.loads(uncompressed) except: return default @anvil.server.callable def get_follower_data(): data = get_data_from_azure(get_blob_service_client(), "NEW_FOLLOWERS.json.gz") return [dict(dt=k, n=len(v)) for k,v in data.items()] @anvil.server.callable def get_unfollower_data(): data = get_data_from_azure(get_blob_service_client(), "LOST_FOLLOWERS.json.gz") return [dict(dt=k, n=len(v)) for k,v in data.items()]
The front end is the visible part of your Anvil application, i.e. the part that will appear in your browser. To start, we just add two
Plots, one for our followers and one for our unfollowers over time to our
Form. Open the
_template.py file in the
client_code/Form directory and add two plot components to the end of the
init_components method like so:
def init_components(self, **properties): ... self.followers_plot = Plot() self.add_component(self.followers_plot) self.unfollowers_plot = Plot() self.add_component(self.unfollowers_plot)
Form/_template.py would look like this:
from anvil import * class Form1Template(HtmlPanel): def init_components(self, **properties): # Initialise HtmlPanel super().__init__() # Set the html template for the app self.html = '@theme:standard-page.html' # Add a GridPanel to the Form self.content_panel = GridPanel() self.add_component(self.content_panel) # Add a FlowPanel to accept NavBar links self.nav_links = FlowPanel() self.add_component(self.nav_links, slot="nav-right") # Add a title to the app self.title_label = Label(text="Bas.codes – Twitter Analytics") self.add_component(self.title_label, slot="title") # Add a sidebar to the app. Comment out the following two rows if you don't want a sidebar in your app. self.left_nav = ColumnPanel() self.add_component(self.left_nav, slot="left-nav") self.followers_plot = Plot() self.add_component(self.followers_plot) self.unfollowers_plot = Plot() self.add_component(self.unfollowers_plot)
Nothing interesting has happened so far. We just created and initialized to member varialbles,
self.unfollowers_plot and set both of them to an empty plot.
Now, it’s time to populate these plots with data. Luckily, these plots are not specific to Anvil, but are good old Plotly graphs.
So, we can simply use the
go functions from the
We can build the followers graph very simple like so:
def build_followers_graph(self): followers = anvil.server.call('get_follower_data') scatter = go.Scatter(x = [x['dt'] for x in followers], y = [x['n'] for x in followers], fill = 'tozeroy', line=dict(color='#2196f3')) self.followers_plot.data = scatter self.followers_plot.layout.title = "FOLLOWERS"
And here, something interesting happens: The first line (
followers = anvil.server.call('get_follower_data')) will instruct Anvil to call the server side function
get_follower_data we just created over the network. This way, your browser does not directly talk to Azure, but to your Anvil server which in turn fetches the follower data from Azure. This is important as you might want to share your Anvil app but not the logic behind it. You can thus easily hide the storage part in the server.
The next lines are pretty straight forward: A scatter plot is created with the data we got from Azure and this scatter plot is assigned to the still empty
Plot object of our
Now, we just have to ensure that our graphs are build when the application is started. We do this by calling our
build_followers_graph method in the constructor of our
For the unfollowers, we repeat the steps above.
The whole code of the
Form/__init__.py file now looks like this:
from _template import Form1Template from anvil import * import anvil.server import plotly.graph_objects as go from datetime import datetime class Form1(Form1Template): def __init__(self, **properties): # Set Form properties and Data Bindings. self.init_components(**properties) # Any code you write here will run when the form opens. self.build_followers_graph() self.build_unfollowers_graph() def build_followers_graph(self): followers = anvil.server.call('get_follower_data') scatter = go.Scatter(x = [x['dt'] for x in followers], y = [x['n'] for x in followers], fill = 'tozeroy', line=dict(color='#2196f3')) self.followers_plot.data = scatter self.followers_plot.layout.title = "FOLLOWERS" def build_unfollowers_graph(self): unfollowers = anvil.server.call('get_unfollower_data') scatter = go.Scatter(x = [x['dt'] for x in unfollowers], y = [x['n'] for x in unfollowers], fill = 'tozeroy', line=dict(color='#2196f3')) self.unfollowers_plot.data = scatter self.unfollowers_plot.layout.title = "UNFOLLOWERS"
If you now run
anvil-app-server --app MyTwitterDashboard --secret AZURE_CONNECTION_STRING=<Your Azure Connection String>
again, refresh your browser at localhost:3030, you will see your follower/unfollower dashboard which should look like this:
Note that you have to export the
AZURE_CONNECTION_STRING environment variable, so that Anvil can pick it up and use it in your server code.
Let’s recap what we have done so far. With only very few lines of code we created a full-fledged web application with a responsive layout (yes, your application will work on your smartphone, too!). On the way, we connected to Azure, processed the results and drew graphs. As if this would not be impressive enough, we can even simplify the whole process even further.
By using the hosted Anvil.works environment, we can interactively design our app. Let’s do that!
If you haven’t already, sign up at anvil.works and log into your account.
First, click on New Blank app and choose Material Design
Next, you should see your Anvil Development Environment which should look like this:
From the Toolbox Menu on the right, drag and drop the Plot component right into the application canvas in the middle
Make sure, that you re-name
Now, head over to the Server Code section in the left toolbox. Click Add new Server Module and paste the server code of the previous section here:
It should look like this:
The last step is to connect our Plot components with our server code just the same way we did it in the Open Source edition of the previous section.
Click on Form1 in the left menu. Switch to the Code view in the middle of your editor. Then, you can paste in the missing parts of your code like so:
Now, there is only one thing left: Our server code does not know about your
AZURE_CONNECTION_STRING. In Anvil you do have Application Secrets instead of environment variables which are specifically designed to store access secrets like our connection string.
You can create an App secret by clicking the
+ Sign on the SERVICES headline in the left toolbox. Now, click on App Secret and create the
Inside your server code, you have to add this line to the top:
As a last step, change the line
AZURE_CONNECTION_STRING = os.getenv('AZURE_CONNECTION_STRING')
in your server code to this line:
AZURE_CONNECTION_STRING = anvil.secrets.get_secret("AZURE_CONNECTION_STRING")
If you now hit the Run button, you should see the exact same app - but this time it’s hosted on Anvil.works.
Running other peoples’ Python code is a very challenging task to do. Anvil has to work out security and performance considerations to provide a secure and responsive platform for everyone. Unfortunately, this means that the free plan of hosted Anvil is restricted to avoid overloading of the platform.
In our sample code we use the
azure module as well as the
gzip module. These modules are not available in the free plan. Luckily, Anvil.works provides a free trial of their professional version. Also, the Open Source version is unrestricted. Should you decide to proceed with your Twitter Dashboard project, you will can always use the Open Source version on your own machine for free.
We have built Azure Functions to Download Data from our Twitter Account, have them processed on the Azure Platform and stored into Azure Blob Storage. Then, we went on and built a frontend for it in Anvil.
All of this happened with nothing else than Python and it did not require us to buy or rent a server. Of course, all the components we discussed are way more powerful than our small example app might make you think. Of course, you can think of extending your Twitter stats dashboard with more
plotly graphs or other frontend elements provided by Anvil.
I hope you’ve enjoyed building a Twitter Follower and Unfollower Dashboard with me! Let me know how this did work out for you and how you want to extend this app. If you like stuff like this, make sure to follow me on Twitter so that you don’t miss future projects we can build together!