|
|
Projects
Overview
This document gives a brief description of some of the projects we've implemented.
Development
Call Centre
Network Management
Misc Applications
Development
Project Management
Description
With several projects on the go at any time, and up to a dozen employees
working on them, it soon became obvious we needed a tool to manage them.
There were several general goals we had for the the Project Management system.
- Visual status of projects
- Impose a methodical approach to all projects and tasks.
- Ensure high priority tasks were being worked on
- Ensure employees always had something to work on
Features
-
Prioritized Project Tree
The project tree lists all projects that are currently on the go. Projects
are prioritized highest to lowest.
-
Project Phases.
Each project is passed through phases: Brainstorming, Planning, Development,
Release and Documentation. At each phase, as much as possible, all tasks are
completed as required by the phase. For example, in the Brainstorming phase all
tasks are brainstormed.
-
Development Mods
Each project is broken down into many tasks, what we call development mods.
Development mods can be new features, bug fixes, hardware setups, purchases,
etc.
-
Methodical Problem Solving
Each task is divided into a prescribed set of stages ensuring the solution
is done in a methodical manner and facilitating peer reviews. The stages
include Problem Definition, Spec Solution, Solution Approval, Assigning to
Employees, Estimating Time, Defining Tests, Coding Solution, Alpha Test, Code
Review, User Documentation, Apply Solution, Beta Test, Documentation, and
Documentation Review.
-
Dependencies
Certain tasks are dependent on others to be completed before they can be
worked on. The project system allows tasks to be made dependent on any number
of other tasks.
-
Critical Paths
The project system helps with time management. For each project a critical
path is determined. The critical path, the branch of dependent tasks that is
estimated to take the longest time, is prioritized thus ensuring the project
will be completed in the shortest time possible.
-
Dynamic Employee Todo Lists
Each employee has a todo list that indicates the tasks assigned to them in
order of priority,. The lists are dynamic. As new projects and tasks are added,
as tasks are completed, the todo lists are automatically reprioritized.
-
Employee Alerting System
Employees are alerted if a task of higher priority than what they are
working on has been added to their todo list. Alerts are made by messenger or
pager depending on how critical the task is.
-
Self-Documentation
All communication regarding each task of a project, including brainstorming
notes, problem definitions, research findings, specifications, questions,
configuration notes, release notes and warnings, and final summaries are stored
with the task using timestamped notes and attachments. The development mod is
the communication vehicle. Questions are written within development mod notes,
and then the development mod is passed to the person the question is directed
to. Thus the full development cycle is documented and available for
reference.
Details of the project development system are documented in the Dev Mod FAQ.
Screenshots
Project Tree
|
Dev Mod Page
|
-- top --
Call Centre
Automated Billing
Description
We designed the billing system for our porchlight.ca Internet service so it
would scale for a large customer base. This involved automated payment methods,
credit card and preauthorized payment, computerized processing and online
services so customers could help themselves. The system can handle thousands of
customers and is completely automated.
Features
-
Online payments
Using an interface to a payment processor, we integrated online payment
services. Customers can select the services they want and pay with credit card
ONLINE. the charges are authorized and processed within minutes.
-
Batch payments
Every month thousands of monthly and yearly renewal charges are processed
using batch systems. Computer programs update customer accounts adding renewal
charges and compiling them in batches. The batches are transferred to payment
processors, two interfaces are used, one for credit card charges and one for
preauthorized payment (void cheque withdrawl). The process is completely
automated and takes about an hour to process several thousand payments.
-
Rejection processing
porchlight.ca implements a pay or no service policy. When a payment fails,
for example, a credit card charge is not authorized or a preauthorized payment
is recalled because of insufficient funds (NSF), service is disabled. Payment
processors make rejection reports available. Our computer programs download
them, parse them, update customers' accounts, adjusting the amounts due and
disabling them, and flag account notices to be available to the customers to
inform them of what happened. This is all automated.
-
Summary reporting
The system includes summary reporting. Reports include daily and monthly
summaries of sales and rejections formatted for easy comparison to bank
statements. Computer programs add general ledger transactions to the
bookkeeping system automatically.
Online Account Notices
porchlight.ca services thousands of customers with only a handful of support
staff. In order to facilitate this, we created an online account notice system,
something we call a proxy server. The online account notice system allows us to
notify customers when their accounts are due for renewal or their credit cards
have expired. The customers respond online. The service saves porchlight.ca
thousands of dollars in support staff wages and postage fees.
-
Redirect to proxy server
The key component of the online account notice system is redirection to
proxy. When a notice is available for a customer, their account if flagged such
that dialup service is redirected to a proxy server. When a customer connects
and uses their browser, regardless of which url they use, http requests are
redirected to the proxy server which servers up a page notifying the customer
that a change is requested. The customer responds to the request, their proxy
flag is removed from their account, and they can resume service as usual. The
system has been ideal for communicating to customers since it is impossible for
them to ignore the notice, and neither can they put off responding.
-
Online notification
We use online account notices to get confirmation of renewal if customers'
paid service is about to expire. We notify customers if their credit card has
expired and request updated info. Account notices can be used to notify
customer of configuration changes, for which we can include online instructions
for them, or to inform customers of changes in policy. While we have customers
addressing a notice, we take the opportunity to get them to update their
contact information. The system allows one or many customers to be flagged with
a particular notice. Customer's get the notice the next time they connect. The
service works 24 hours a day, 7 days a week. If there are several notices, they
can respond to all of them in one sitting.
-
Online activation
Each customer activates their dialup service account using an online account
notice. The notice outlines the billing procedures explaining the amounts and
dates that charges will be applied and includes the service terms and
conditions. The customer is unable to activate their account without viewing
and agreeing to the terms. This method has gone a long way to resolving billing
disputes as everything is presented up front and customers cannot claim
ignorance. In order to view the activation account notice, the customer has to
use their own computer and connect to our service. This acts as a test that
their equipment is compatible and configured properly. Without activation,
customer's aren't charged. They only pay for terms they agree to and for
service they can use. If customers choose to stop porchlight.ca service, their
account is available for reactivation at any time. The customer need only dial
into the porchlight.ca server and they we will be prompted with an activation
notice. Activation is available 24 hours a day.
Screenshots
Online Account Notice
|
-- top --
Asterisk PBX
Description
Our support centre requires a phone system of some sophistication, and the
one we were using had limited features with no hope of customization. It was
disappointing considering the high price we paid for it. We were excited to
hear about Asterisk, the open source PBX. Asterisk has gone through some
growing pains, but we have been able to make good use of it.
Although Asterisk works well as a PBX, it's real power is as a VOIP gateway.
Asterisk has the ability to bridge the telco and IP networks. Currently all our
office phones and tech support services use VOIP.
Using Asterisk we've cut long distance costs and we have a full featured
phone system using inexpensive hardware and software.
Asterisk Features
-
Standard PBX functionality.
We make use of many Asterisk standard PBX features within the office: call
queuing, call recording, voicemail, conferencing, music-on-hold,
auto-attendant, and dynamic Interactive Voice Response.
-
Call Detail Records
We log the details of every call in a MySQL database. This makes producing
statistical reports automatically very easy.
-
Call routing.
Asterisk allows complete flexibility over routing phone calls.
ip -> ip
ip -> telco
telco -> ip
telco -> ip -> telco
Customization Features
Asterisk Gateway Interface (AGI) allows custom functionality to be added with
many programming languages including Perl, the scripting language we use. Using
this feature we've been able to add some custom features.
-
Identify callers
Customers are prompted to enter an account number (their phone number)
allowing us to identify them. When the call is directed to an support agent,
the customer account information is brought up on their monitor
automatically.
-
Dynamic agent status
The status of support agents is changed from within our existing interface.
Agents log in and out of our support interface and their status in the phone
system is updated automatically.
-
Prioritize callers
We prioritize callers in wait queues based on past usage. Customers who
require many time consuming support calls are given lower priority over
customers that use little support time.
-
Real-Time Monitoring
We've created our own monitor that displays real time stats related to the
the phone system. Reported are queue stats including the number of agents in
each queue, average wait time and number of abandoned calls. Individual agent
stats include status, time on current call, average time on calls, total time
on hold, and total time unavailable.
-
Quality Control
We record all calls. By reviewing them, managers can determine where
employees can improve their support for customers. Occasionally the recordings
can be used to resolve disputes where customers claimed they called in the past
and said such and such and requested such and such.
-
Asterisk call checking
We created self check scripts whereby asterisk calls itself to make sure
services are running properly. Nagios alerts personnel if the calls don't get
through.
VOIP Services
We extended our VOIP services so they could be used by external parties.
porchlight.ca provides local DIDs and VOIP termination service.
-
Local DIDs and VOIP termination
porchlight.ca makes available DIDs local to major calling areas in
Southwestern Ontario. With termination services, contacts in Southwestern
Ontario can connect to VOIP servers worldwide with a local call.
-
Online account maintenance
In line with our philosophy of making services convenient for customers and
low-maintenance for us, we created a web interface for account maintenance.
Customer's can create an account, select DIDs from whichever calling areas they
desire, and pay for minutes online. Customers purchase minutes in bulk and can
use them with any of their DIDs. The interface includes reporting. Account
statements, call details, and purchase history are available. The system
includes configurable reminders and alerts so customers can be notified when
minutes on the account are low.
Screenshots
Asterisk Real-Time Monitor
|
VOIP centre
|
References
Asterisk web site: http://www.asterisk.org
-- top --
Call Centre Pages
Description
Our goal was to get as many customers as possible using porchlight.ca dialup
service. However, we wanted to maintain quality customer support. In order to
do this we provided our call centre agents with state of the art tools. We
developed a call centre agent interface, a web page application with all the
features the agent would need to provide customer account maintenance and
support.
Features
-
Customer account maintenance
The primary purpose of the call centre agent interface is for account
maintenance. Call centre agents can create new accounts and update existing
ones. Everything related to the account, contact information, account type,
renewal dates, purchase history and billing information is available for review
or editing.
-
Comments
In order to provide quality and orderly support, we track every
communication and every change made to the account from it's inception. Each
account page makes available the complete history of comments from the account
so any call centre agent can quickly become familiar with the status should a
customer call for support.
-
Phone system integration
The call centre pages are integrated with the Asterisk PBX phone system.
When a customer calls porchlight.ca support line, the system identifies the
customer and triggers the account to be displayed in the agents browser prior
to transferring the call to the agent. The agent is thus prepared to deal with
the issue quickly and efficiently. Phone conversations are recorded and
available to the agent from within the call centre pages. If a dispute arises
from a previous call, the agent can resolve the issue themselves without
delay.
-
Technical support
To help troubleshoot technical problems with customers services, the call
centre pages provides tools and system logs. The agent can test the account
connectivity, run diagnostics, eg ping tests, and check system logs filtered on
the account in question. The agent has access to a tech support wiki doc.
Solutions to every problem agents has ever faced are documented. The wiki
format makes it possible for agents to update the documentation as
required.
-
Interactive interface
The call centre agent interface is interactive. Our system can pop up web
pages on the agent computer. This is useful for delegating return calls or
follow ups to agents. The system can determine which agent is not on call and
available. Return calls can be scheduled at the request of the customer. The
system will initiate the return call at the scheduled time. The interactive
interface makes it possible to distribute employee bulletins and get
confirmation from employees regarding their future work schedules and past
hours.
Screenshots
Agent interface tools
|
Customer account page
|
Agent interface bulletin
|
-- top --
Human Resource Management
Description
porchlight.ca has maintained a staff of 15-20 people over the years. Running
a call centre inevitably involves some turnover. We've devised some tools and
methods to simplify hiring, scheduling and payroll.
Features
-
Employment aptitude tests
porchlight.ca uses aptitude tests to screen applicants for positions. The
number of people applying for positions can be huge and the aptitude tests give
us a way to whittle the number down to those suitable for an interview. The
testing system was designed in-house. We include a link in job ads to a page
that provides a full job description and an application form. Applicants
provide contact information and schedule a time for the aptitude test. At the
office several computers are set aside for testing. The testing system is
self-contained. It provides instructions and multiple choice questions. Some
positions required several levels of testing. The system grades the first test
and determines if the applicant is eligible for the next level. If so, the
system begins the next level of testing or schedules a time later to conclude
the testing if it is not convenient for the applicant. The system is scalable
and can manage hundreds of applicants in a short time with minimal manual
input. We have stats from all applicants and all test questions. This allows us
to determine if questions are effective and we can fine tune the tests if
necessary.
-
Scheduling
In order to assure there is sufficient support staff available at all times,
we've created a tools to simplify scheduling. Schedules are created for each
employee for the coming two weeks. Schedules can be adjusted using a web page
interface. Hours can be shortened or extended as necessary, and shifts can be
switched between employees. We have reports that display who is scheduled at
any given time during the week and pages that facilitate vacation planning.
Call frequency reports help us predict call volume so we ensure we have enough
support people scheduled. We created a "soft" punch clock. Employees sign in
and out as they come and go by clicking a web page on a computer near the
employee entrance. The exact time of their entrance and exit is recorded and
used to determine their hours worked.
-
Payroll
We do all payroll in-house using software we developed. The system
calculates gross pay, vacation pay, holiday pay, deductions and net pay for
each employee. The system can handle lump sum payments and compensation for
employee mileage. Pay is deposited directly in employees' bank accounts. They
access paperless pay stubs online. General ledger transactions are updated in
the bookkeeping system automatically. We have programs to calculate Record of
Employment values and create T4 statements.
Screenshots
Aptitude Test Schedule
|
-- top --
Network Management
Console Servers and Remote Power Control
Description
With servers and other equipment running from remote sites, we found we were
spending a lot of time travelling to do maintenance. Several times we had to
travel an hour to a location in order to power cycle a machine, and then
turn around and ride another hour back. Something needed to be done about that.
We installed console servers and remote power control. This allowed us to
manage and troubleshoot remote servers from the comfort of our own desks.
Features
-
Remote configuration of devices
We set up all network routing equipment, switches, routers, etc. so they can be configured remotely.
-
Server output to console.
We configured all Linux boxes to redirect screen to console via serial.
Through the serial console output we can view the server boot messages and boot
different images using GRUB. (e.g. normal boot, tftp rescue boot, or even tftp
reinstall boot).
-
Remote power control via Cyclades AlterPath PM10-L30A
We installed Cyclades AlterPath power bars at each remote location. Each
power port on these power bars can be independently controlled, so we can power
cycle any single device plugged into it. Power ports can be assigned names,
which simplifies organizing and identifying devices.
References
Cyclades Terminal Server - Manual (pdf)
Cyclades Remote power control - Spec sheet
Sample GRUB configuration file: grub.conf
-- top --
Nagios
Description
porchlight.ca's entire network infrastructure is monitored in realtime using
the open-source Nagios system. Many of our services are required 24/7 and it
is critical that we not only keep the servers running smoothly, but also ensure
that the services running on the servers are operating correctly.
Features
-
Monitoring network hosts.
All servers on the network are monitored. Nagios alerts personnel whenever a
server is registered as down. Equally important, Nagios alerts people when the
server is back up to inform everyone things are back to normal.
-
Monitoring network services.
All services required for smooth operation are monitored. These include
those used by customers as part of their Internet service, radius
authentication, email services, personal web page services, and Member Centre
services, and services needed for employees to provide support and development,
httpd, mysqld, and asterisk. Nagios provides alerts if any of these are not
operating properly.
-
Monitoring resources.
All system resources are monitored. For example, Nagios alerts if network
bandwidth utilziation is abnormal, hard drives are low on disk space, or a
server's overall health is not well.
-
Alerting
Nagios sends critical alerts via instant message and pager. Warning alerts
are sent by instant message and email.
-
Customization
Nagios is very flexible. The plugin design allows us to create our own host
and service checks. In some cases we not only test that a services is running,
but also test that it is functional. For example, we test that an httpd server
is working properly by accessing a web page on the server. We test that
asterisk is functioning by periodically using it to call itself, testing both
incoming and outgoing call services.
-
Nagios Self Check
We use Nagios to monitor itself. It's critical that Nagios be in operating
order since we've come to rely on the absence of alerts to indicate all systems
are running normally.
Screenshots
Nagios screenshots: http://www.nagios.org/about/screenshots.php
References
Nagios website: http://www.nagios.org
-- top --
Redundant Failover Systems
Description
Early on we decided each server should have an identical backup. In order to
minimize down time, we implemented two-computer failover clusters for each
server. This provides hardware and software redundancy, permits maintenance on
servers with little interruption, and simplifies beta testing new services and
upgrades, all the while using consumer-grade hardware.
Features
-
Hardware and software redundancy
With two computer failover clusters, if the primary server experiences a
hardware malfunction or a service on it dies, the cluster fails over to the
secondary server. Services provided by the server continue to be available.
-
Availability during maintenance
The failover cluster made it very convenient to perform periodic maintenance
and upgrades on servers with minimum interruptions. While the primary server is
providing services, the secondary server can be upgraded. Once complete, the
server is failed over and the secondary server takes over providing the
services. The primary server than can be upgraded.
-
Beta testing fail back
When implementing changes in services, the new system is tested by bringing
down the primary server and testing the changes on the secondary server. If
there are any issues the primary server is quickly brought back up and the
existing configuration is used. This minimizes service interruptions to our
customers. A server can be beta tested for a good period of time (eg. one
week) and if we're satisfied, we upgrade the primary server with the new setup
by syncing the new server image to it. The two servers are identical and the
failover cluster once again provides redundancy.
-
Consumer-grade hardware
Using the Red Hat Cluster Suite, we were able to create failover clusters
using consumer-grade hardware. Costs were very reasonable, technicians were
familiar with the setup, and replacement parts were the same ones we were using
in desktop computers.
References
Red Hat Cluster Suite: Configuring and Managing a Cluster
-- top --
Stateless Linux
Description
In order to reduce the amount of time spent on maintenance, porchlight.ca
has implemented a stateless Linux server set up as designed by Fedora Core.
Features
-
Stateless Images
Whenever upgrades are needed, an image is updated and any number of
computers based on that image are updated on the fly.
-
Rollback
If a current image has problems, the sys admin can quickly rollback to the
previous image. All computers based on the image are quickly updated.
-
Simple Computer Additions and Replacement
Adding or replacing computers on the network is simple. Format the hard
drive, load up the image, and you have a fully configured working computer good
to go.
-
Read-only Root
System files of the OS are mounted read-only. This protects a system from
human error and prevents exploits and viruses from having any affect. The sys
admin is confident no core/root files have been altered.
-
Central Management
Using stateless scripts it is possible to modify a 'stateless filesystem'
from a remote computer and rsync the changes. A network of hundreds of
computers can be upgraded remotely.
References
Fedora Projects: Stateless Linux http://fedoraproject.org/wiki/StatelessLinux
Introduction to Stateless Linux
-- top --
Misc Applications
Bookkeeping
Description
We found bookkeeping software programs were not flexible enough for us. It
was difficult to import data and we couldn't customize to our liking. So we
built our own. The system is used for business and personal finances.
Features
-
General accounting features.
Our bookkeeping system is very basic but it has all the features required
for general accounting. The system includes chart of account maintenance,
general ledger entry and inquiry, cheque writing, fincancial statements (trial
balance, profit and loss and balance sheet) and year end processing.
-
OFX Import
One must of our system was to the ability to import our bank statement data.
We choose the OFX format since it is common and relatively well documented. OFX
files can be downloaded from banks' web site and imported into our system.
Statement descriptions (generally vendor names) can be associated with specific
ledger accounts and applicable taxes can be assigned, such that the import
creates double entry transactions automatically. This saves a ton of time and
reduces data entry errors.
-
MySQL database backend
Software programs that use propriety data formats make it impossible to get
at the data for manipulating or importing. Our system using MySQL. The data is
readily accessible. The bonus is we can import general ledger entries from
other systems. Currently transactions for dialup Internet billing, billing
rejections, refunds payments and payroll deposits are made directly to the
bookkeeping system. Automation saves time and limits errors.
Screenshots
OFX Import
|
-- top --
Reporting
Description
We love collecting data and producing stats. With the wide range of activity
from customers, from employees, from promotions, from servers and automated
systems it can make your head spin. Reports with numbers and graphs are
comforting as they reduce the chaos to something simple, orderly and
understandable. Computers make it relatively easy to collect, store, compile
and report data and so we take full advantage fo them.
Screenshots
Online Stats
|
Signup Poll Stats
|
-- top --
|
|
