Personal Weather Station
A weather station based on the Raspberry Pi
7. Web page generation Perl script
Now that we have a log file that contains the current outdoor temperature and humidity we need to have a method to display these values. On most commercial weather stations, a simple LCD display shows the current temperature and humidity. Our weather station however, has a log file that has an extensive history of the outdoor temperature and humidity. Using this history we can display far more information than just the current temperature and humidity.
The Perl script that we create will generate a html file which will be placed in the Raspberry Pi's web server directory. When we visit the Raspberry Pi's IP address using a web browser, either on our laptop or smart phone, it will display the web page that we created.
Since we have a long history of temperature and humidity data, we will start by displaying the weather data on the web page in the following ways:
- Current temperature and humidity updated every 15 minutes.
- Maximum and minimum temperatures in the last 24 hours and the times they occurred
- A graph of temperature and humidity over the last 24 hours.
The Perl script will be called process_weather_data.pl.
7.1 Parsing the log file
The Perl script will need to first read in the data from the log file. The log file is continually growing in size as the receive_remote_sensors.pl Perl script writes data to it from the remote sensors. This can cause the process_weather_data.pl script to spend a lot of time searching the file for the last 24 hours of data, especially when there is almost a year's worth of data in the log file.
To speed this process up, we can use a system call from the Perl script to the Linux command tail. Tail is a Linux system command that shows the last few lines in a file. When we add the option -n to tail, we can specify how many lines at the end of the file to show.
Since our log file has entries every 5 minutes, or 12 entries every hour, we need the last 288 lines from the log file (24 hours * 12 entries per hour ) . A system call to tail with the output re-directed into a temporary file will make parsing the log file easier.
system("tail -n 288 /home/pi/weather/weather.log > /tmp/weather24hr.log")
Next we can read this much smaller file /tmp/weather24hr.log that only contains the last 24 hours of weather data. The log file weather.log contains data in the following format:
Thu Mar 12 21:35:05 2015,-4.4C,58.3%; Thu Mar 12 21:40:19 2015,-4.6C,59.3%; Thu Mar 12 21:45:04 2015,-4.7C,59.0%;
The Perl script will loop through the file and read the data into three arrays; time stamp, temperature and humidity. As the data is read into the three arrays, we strip off the units so that we have the raw data values stored in the three arrays; @timestamp, @outdoortemp and @outdoorhumidity.
7.2 Computing maximum and minimum temperatures
While the Perl script reads in the temperature and humidity data from the log file into the arrays, we can use this opportunity to calculate the maximum and minimum values.
Before starting to read the log file, we initialize $maxtemp to -40C and $mintemp to 100C. By setting the maximum value initially to the lowest possible temperature value, we can almost guarantee that the first value read from the log file will be greater than -40C. If we had initialized the $maxtemp variable to 0 as we do for most other variables, we would never detect the maximum temperature on a winter's day when the temperature may never get above freezing (0C). If $maxtemp was initialized to 0 and on a cold winter's day the temperature only reached -10C , the program would erroneously report 0 as the maximum temperature for the day when in fact the maximum temperature was -10C.
As we read each value in the log file, we compare it to the maximum value. If the current temperature reading is greater than the saved maximum value, we update the maximum value to be equal to the current temperature value. At the same time, we save the time stamp from the current temperature reading as that is the time at which the maximum temperature occurred. A similar method is used to find the minimum temperature except that the program needs to check if the current temperature is less than the recorded minimum value. Figure 7.1 shows an example of calculating the maximum temperature while reading the data in the log file into an array.
while(<WLOG>) { /^(.*),(.*),(.*);/; $currenttimestamp = $1; $currentoutdoortemp = $2; $currentoutdoorhumidity = $3; $currentoutdoortemp =~ s/C//; $currentoutdoorhumidity =~ s/\%//; if (!($currentoutdoortemp eq "")) { if ($currentoutdoortemp > $maxtemp) { $maxtemp = $currentoutdoortemp; $maxtemptime = $currenttimestamp; } if ($currentoutdoortemp < $mintemp) { $mintemp = $currentoutdoortemp; $mintemptime = $currenttimestamp; } } $outdoorhumidity[$index] = $currentoutdoorhumidity; $outdoortemp[$index] = $currentoutdoortemp; $timestamp[$index] = $currenttimestamp; $timestamp[$index] =~ /([a-zA-Z]*) .* ([0-9]*:[0-9]*):.* /; $timestamp[$index] = "$1"." "."$2"; $index++; }
Figure 7.1 Reading the log file and finding the maximum temperature
7.3 Generating graphs
The Perl script will also generate graphs of the last 24 hours for both outdoor temperature and humidity. A graph can display interesting trends with the temperature and humidity. We can see how quickly the temperature falls on a cold night, or how the humidity rises before a thunderstorm.
To generate these graphs, we will use a Perl module called GD::Graph. GD Graph is an open source Perl module that takes in an a set of arrays that contain the X and Y values for the graph. It plots the data and generates an image of the graph that you print to a file. The result is a png image file that contains the graph. We will then reference this image file in the html of our web page to display the graph.
Before we generate the graphs, we need to fix one small issue with the data stored in the temperature and humidity arrays. Remember that in our receive_remote_sensors.pl script, we decided to null out any temperature or humidity readings if we did not receive updated data from the remote sensors. So there is the possibility that in the data arrays, there are some null values where data from the sensors is missing. On the graph we want this missing data to be displayed as a discontinuous line so that the line on the graph appears broken whenever we are missing data from the remote sensors.
We can achieve this by setting the graphing function to skip undefined values. However, our arrays contain null for missing data and we need to change this to an undefined value for GD Graph to display the discontinuous line correctly. If we leave the null values in the array, GD Graph will assume those values are 0 and draw a line down to 0 on the graph instead of leaving a discontinuous line.
We need to map each null value to the Perl undefined value (undef). We can do this by using the Perl function map.
@outdoortemp = map {$_ eq "" ? undef :$_} @outdoortemp;
@outdoorhumidity = map {$_ eq "" ? undef :$_} @outdoorhumidity;
Next the parameters for the graph need to be set. These parameters include basic settings such as the title for the graph and the labels for the X and Y axis. Figure 7.2 shows the basic settings for the temperature graph.
my $graph = GD::Graph::lines->new(800,400); $graph->set( long_ticks =>1, skip_undef =>1, x_label => 'Time', x_ticks =>0, x_label_position =>0.5, x_labels_vertical =>1, x_label_skip => 12, y_label => 'Temperature (C)', title => '24 hr Temperature Trend') or die $graph->error;
Figure 7.2: GD Graph settings for outdoor temperature graph
There are a couple of important settings in figure 7.2 that we need to make our graph look nice. Setting long_ticks to 1 causes a line to be drawn for each tick interval on the graph. By setting x_ticks to 0, we turn off vertical lines on the graph, but leave horizontal lines. This makes it easier to visually see the different temperature points on the line.
By setting x_label_skip to 12 , GD Graph will only label every 12th point on the X axis. Since the X axis is time, this will only show the labels at an hourly interval. If we did not set this, GD Graph would label each data point on the X axis, all 288 of them. This makes the X axis labels look to cluttered. Setting x_label_vertical to 1 causes GD graph to print the labels vertically rather than horizontallyi.
Setting skip_undef to 1 causes the graph function to not draw any points when the value is undefined. Remember, we set any missing temperature or humidity values to undefined using the Perl map function. Skip_undef is what makes the graph function draw the discontinuous line when data from the remote sensor is missing.
GD Graph requires an array of arrays to be passed to it as the data set. The array in index 0 is the X axis values, and the array in index 1 is the Y axis values. If you want to plot a second line on the graph , you would put that in index 2. Since we want time on the X axis and temperature on the Y axis, we will load our time stamp array in to index 0 and our temperature data set array into index 1.
@OutdoorTemperaturedat[0] = [@timestamp];
@OutdoorTemperaturedat[1] = [@outdoortemp];
Next we pass this array of arrays to the graph function.
my $gd = $graph->plot(\@OutdoorTemperaturedat) or die $graph->error;
This generates the graph and stores it in memory. The final step is to print the graph to a file using the png image format. This image will be referenced in our html file to display the graph on our web page.
open(IMG,'>/var/www/image1.png') or die $!; binmode IMG; print IMG $gd->png; close(IMG);
A similar method is used to generate the humidity graph. An example of the resulting graph is shown in figure 7.3.
Figure 7.3: Sample 24 hour outdoor temperature graph generated by GD Graph
7.4 Creating the web page
The final stage in process_weather_data.pl is to create the html file for the web page. When a web browser goes to the IP address assigned to the Raspberry Pi, the Raspberry Pi will serve the file /var/www/index.html as the default web page. We will create this file as our weather stations main html page. Later we will reference other html files from this main file.
The html file contains basic html tags which can be styled by the included style sheet style.css. Style.css contains basic formatting for the web page. In later sections we will go into detail on how to improve the look of the weather stations web page.
The Perl script prints the basic html tags and includes the current temperature and humidity. It also prints into the html file the maximum and minimum temperatures and the times at which the maximum and minimum temperatures occurred.
An image tag is created to reference and display the graphs of the 24 temperature and humidity trends that we created earlier using GD Graph.For now the web page looks plain and simple, however later we will add additional styling and features as we add more remote sensors.
7.5 Final web page result
Now that the Perl script that processes the weather data and generates the web page is complete, we need to set it to run every 15 minutes. The log file weather.log is continually being updated every 5 minutes by receive_remote_sensors.pl which is running in the background. A reasonable update frequency is about once every 15 minuets. This means that we need to re-run process_weather_data.pl every 15 minutes to read the new weather data and re-generate the graphs of temperature and humidity as well as create an updated web page.
The easiest way to do this is to set the Linux system process cron to run process_weather_data.pl every 15 minutes. Open a Linux terminal on the Raspberry Pi and run:
crontab -e
Next add the following line:
*/15 * * * * /home/pi/weather/process_weather_data.pl
This will cause the Raspberry Pi to run process_weather_data.pl every 15 minutes automatically and update both the graphs and the web page with the latest temperature and humidity data.
Figure 7.4 shows an example of the weather station;s web page generated by process_weather_data.pl.
