Tag: Linux

How the Web Was Won: Part 5 – Maturing

By the early 2000s, the Web no longer felt like an experiment. It had become business infrastructure. Websites were expected to work around the clock, online services were becoming essential, and organisations increasingly depended on systems that had once been regarded as little more than interesting technology.

Around this time we introduced Rhythmyx, one of the first commercial content management systems I worked with. It brought new capabilities but also a new way of thinking. Development became more structured, marketing expectations grew, and the pressure to deliver increased. The Web was no longer simply about building useful tools; it had become a business in its own right.

One Christmas we carried out a major deployment just before the holiday. As anyone who has worked in IT will recognise, software has an unfortunate habit of revealing its most interesting bugs only after everyone has gone home. My colleague Peter spent much of his Christmas break trying to resolve unexpected problems. It was one of those moments that reminded us how much responsibility the Web now carried. What had once been static pages viewed by enthusiasts had become systems on which organisations depended.

Around this time I noticed a change. In the early days, “web development” meant doing a little of everything: HTML, graphics, servers, databases, networking and whatever else was needed. As organisations grew, those responsibilities became divided into specialist roles. Interface designers, web designers, multimedia developers, database administrators and network engineers all became experts in their own fields. The advantage was deeper expertise. The downside was that fewer people understood how the whole system fitted together. Increasingly, my role became one of connecting those pieces and helping trace problems that crossed the boundaries between teams.

A good example was the familiar “500 Internal Server Error”. A web designer might assume it was a server problem. A server administrator might suspect the database. A database specialist might look for faults in the application. In reality, the cause could lie almost anywhere: a configuration change, a permissions problem, a missing library, a database connection or an application bug. Solving it often depended less on deep knowledge of one component than on understanding how all the components interacted. In our case, after two weeks, our best developers were stumped. So I stepped in and suggested they check the server was correctly reporting the error. In fact, it was a brand new server and had been misconfigured – the problem was solved.

During these years I also found myself thinking more about the future than the present. I had become interested in ideas such as electronic currencies, including an early proposal known as eCU (see footnote), which suggested that money itself might one day exist primarily as information moving across networks. At the time it felt speculative. Looking back, it now seems less remarkable than it did then.

Not long afterwards I was asked to write a paper for senior management exploring how I thought the Web might develop over the following decade. Alongside international collaboration and changing patterns of scientific publishing, I briefly considered the possibility that advances in artificial intelligence might arrive sooner than many people expected. I had no particular insight into how it would happen, only the feeling that computing was beginning to move beyond simply storing and retrieving information towards generating new ideas. Whether I was right or wrong mattered less than the exercise itself. One of the privileges of working in technology is that you are constantly encouraged to think a little way beyond the horizon.

Around 2009, I briefly met Tim Berners-Lee, creator of the World Wide Web (not the Internet, as some mistakenly believe) and Jacob Nielsen, named “guru of Web page usability” in 1998 by The New York Times, Tim was engaged in a group conversation in an Institute foyer when I joined the conversation without noticing at first who he was. I met Jacob at a conference on the future of usability on the Web.

Looking back now, what strikes me most is not the technology itself but the pace at which it changed. During my career I watched the Web evolve from hand-written HTML pages uploaded over dial-up connections into the foundation on which businesses, publishers and scientific organisations increasingly relied. The excitement of those early years gradually gave way to reliability, security, accessibility and scale. The Web had grown up.

People often remember the early Web through the companies that became household names. I remember it through FTP sessions, scanned photographs, frozen screen locks, millions of web pages, inaccessible HTML, late-night deployments, and the quiet satisfaction of making things work.

Five in the Morning

The photograph below was taken at around five o’clock one morning during software testing at the Institute of Physics, towards the end of my career. Looking at it now, I’m struck by something I hadn’t noticed at the time. After more than twenty-five years working on the Web, I was still curious enough to turn up before dawn just to make sure everything worked properly. Technologies had changed beyond recognition since my first HTML page, but the fascination hadn’t. I think that’s what the photograph really captures.

You can always find a full index of all the articles on the page under About me in the main menu.

Web Stories

The Tunnel of Forgotten Macs
During the water leak before the comms room move, the Institute’s Facilities Manager led me down two floors below ground level one day and along a passage to some store rooms. But we passed a door that I hadn’t noticed before, so I asked where it led. She said I could take a look, so I did. Lined up along an empty passageway were around eighteen redundant Bondi Blue iMac G3s. Once they had been among the most distinctive computers in the world; now they sat forgotten beneath the building, like an unexpected museum exhibit from the early Web.

eCU Footnote

From Google
ECU (often stylized as eCU) was an early centralized digital currency and electronic gold-backed project that operated during the early 2000s tech boom. It modeled itself after the original European Currency Unit but was designed for internet commerce. Why It Failed So Quickly. The Gimmick: The project sold physical, transparent acrylic promotional tokens embedded with a holographic security strip or micro-serial number to simulate “tangible digital money.” The Collapse: Unlike modern decentralized cryptocurrencies, it relied completely on a centralized issuer. Amid the regulatory crackdown following the Patriot Act of 2001—which targeted early digital gold currencies like e-gold for lacking KYC/money-laundering compliance—the project panicked, ran out of capital, and went defunct within a matter of months.

How the Web Was Won: Part 4 – Systems

By the late 1990s the Web was growing up. Companies no longer wanted just a few attractive web pages; they wanted systems that could manage large amounts of information, allow staff to update content themselves and remain reliable enough to serve thousands of visitors every day.

I attended a Network Building course near Old Street, because one of my first tasks at the Garrick Corporation had been to install a peer-to-peer network and Eudora email system, not knowing anything about managing such systems. Both operated well, but I knew I had been lucky – things could have gone badly wrong – so I wanted to be prepared the next time. I also continued with some contract work and applied for a few full-time positions.

I was offered three different jobs in London. One was with a web design company, another was with a commercial media organisation, and the third was with the Institute of Physics. Although the salary was lower than the others, the Institute appealed to me because it seemed to offer the greatest opportunity to learn and experiment. Looking back, it was probably the most important career decision I made.

The Institute’s web infrastructure was based at the Institute of Physics Publishing offices in Bristol on Linux, Apache, Oracle and MySQL (later, PostgreSQL), using Perl to generate dynamic web pages. Until then I had never written Perl, and I had certainly never used the vi editor. Fortunately I was surrounded by some exceptionally talented colleagues who were always willing to share their knowledge. Peter, the go-to Perl guru, in particular, patiently introduced me to Perl, mod_perl and the way the Institute’s publishing systems worked. Before long I was writing production code myself.

One of my first major projects was to design and build what became known as the BDG Tools – an online system used by the Institute’s branches, divisions and groups to maintain their own web pages. Today it would simply be called a content management system, but at the time very few organisations had anything similar. The aim was to allow physicists around the country to update their own information without needing to understand HTML or rely on the web team for every change.

The first version worked well, but it also revealed something important about human nature. The different groups soon discovered that by changing the parameters in the page URL they could view and even edit each other’s pages. Most of it was done in good humour, with one group quietly correcting another group’s mistakes, but it was obvious that the system needed tighter controls. I quickly redesigned it so that the important information was hidden from the URL and checked securely behind the scenes. It was one of my earliest lessons that, if users can find an unexpected way of using a system, sooner or later they will.

When the system finally went live it proved remarkably successful. Instead of maintaining hundreds of static pages ourselves, responsibility was shared across the organisation while the design and navigation remained consistent. It also taught me another lesson that has stayed with me ever since: reliability matters far more than cleverness. Users soon forgive a system that looks ordinary, but they rarely forgive one that fails when they need it.

While all this was happening, in 2000 the Institute of Physics Publishing received the Queen’s Award for Enterprise in International Trade, recognising its success in exporting British scientific publishing around the world through innovative use of web technology. I attended the Award ceremony, but I was only a small part of a much larger team, responsible primarily for the user interface and front-end systems. It was rewarding to know that the work we were doing contributed, in however small a way, to that achievement.

Not long afterwards another challenge arrived. New accessibility standards from the World Wide Web Consortium (W3C) meant that large websites had to become much easier to use for people with visual impairments and other disabilities. My manager asked me to lead the initiative to bring the Institute’s website up to the highest practical accessibility standard. With six million pages, this was never going to be a quick task. I visited the RNIB, spoke with accessibility specialists and consulted the W3C guidance before drawing up a strategy. Achieving the very highest level proved unrealistic with the technology available at the time, but over the following years we successfully brought the site up to the second-highest standard. It was painstaking work, but it fundamentally changed the way I thought about designing websites. Accessibility was no longer an optional extra; it became part of good engineering.

Around 2002 the Institute purchased the neighbouring building, and one of the projects arising from the move was relocating the communications room. I volunteered to manage the migration. It was far larger than anything I had previously undertaken, involving Siemens engineers, relocating the telephone exchange and drilling large service ducts through the foundations linking the two buildings. Everything had to be carefully planned to minimise disruption, because the Institute’s communications depended on it. The move was completed successfully over a weekend, the only outage reported being one person’s loss of network through one overstretched Cat 5 cable in the huge bundle running between the two buildings. I received a Highly Commended award for the project. Shortly afterwards I was promoted to IT Manager (officially: Online Services Manager).

One unexpected challenge was highlighted prior to the preparations during an installation of new servers. I arrived one morning to find water gushing from the wall onto one of the machines. The facilities manager explained that the building had a hidden world beneath it: old waterways, tunnels and pumps that kept the foundations dry. A failed pump had caused the leak. The problem was fixed quickly, and the server survived.

My network training came in very handy for supporting the complex connections between London and Bristol, and earlier experiences configuring Eudora email for the Garrick Corporation were useful when I migrated our existing email system’s account to Eudora using as my guide only a bulk load method described in the back pages of a badly translated manual. As part of my IT Manager role, I finally became responsible for acquiring, administering and configuring DNS records for our many domains.

Looking back, this was the period when I stopped thinking of myself simply as someone who built websites. I had become responsible for systems, infrastructure, projects and people. The Web was no longer a fascinating new technology on the edge of computing; it had become part of the organisation’s core business, and I found myself helping to build the foundations on which that future depended.

Part 5

You can always find a full index of all the articles on the page under About me in the main menu.

Web Stories

Around 2002, I stayed with friends in Cyprus who had two young children. One afternoon I showed them the basics of HTML. We typed a few simple tags, saved the file, refreshed the browser, and suddenly there were colours, headings and pictures appearing on the screen.

Their reaction was unforgettable. They stared at the monitor in complete amazement. The idea that they could type a few lines of text and create something that anyone in the world might one day see seemed almost magical.

I remember feeling exactly the same when I first discovered programming years earlier. Technology changes, but that sense of wonder never really goes.

As far as I know, one of those children later became a web developer. Whether that afternoon had anything to do with it, I’ll never know, but I’d like to think that, for a moment at least, they experienced the same excitement that had first drawn me into the Web.

How the Web Was Won: Part 3 – The Boom Begins

After teaching myself HTML, Photoshop and FTP, and experimenting with my own pages on the early Web, I started jobhunting. After answering 10 adverts in The Guardian without success, I saw a tiny advert for a web developer in the Evening Standard. It did not look particularly interesting, but a friend persuaded me to try anyway. When I arrived at the company, I discovered that they had recently paid for a website to be designed but could not get it working properly on their PCs. I looked at it and immediately saw the problem: they were using Internet Explorer 2, which did not support frames. I downloaded Internet Explorer 3, which solved the problem, and they were so impressed that they asked me what salary I wanted. The advert had offered £16,000. I suggested £20,000. They laughed and offered £18,000. I replied with £18,500, and they agreed. That was how I became a professional web developer.

The company was a small Chelsea Wharf start-up, calling themselves the Garrick Corporation. The atmosphere was energetic and sometimes chaotic. Everyone suddenly wanted a website. Technical people were discovering what was possible, while salespeople were discovering that companies would pay for websites even if they did not fully understand what they were buying. Some of the sites were very simple. We were sometimes building websites for around £150, and at the peak we could produce three or four small websites in a day. Looking back, it seems extraordinary how quickly the industry moved from hobbyists creating pages at home to companies selling web development as a commercial service.

Among the projects I worked on were Brighter Pictures and Harkness Sayers, both movie business production companies. These were still hand-built websites, created before content management systems became common. Pages were designed individually, graphics were created in Photoshop and everything was uploaded manually using FTP.

I preferred manually coding HTML, which gave cleaner code, but for big projects or tricky configuration, I sometimes used Dreamweaver or HotMetal Pro. Graphics were created in Photoshop, while the finished pages were uploaded one by one using FTP. There were no content management systems, no templates (other than those I made) and no drag-and-drop website builders. Every page was built by hand.

The National Trust project was probably the biggest website I worked on at that stage. They already had a small website, but they had not attempted to put their properties online in any systematic way. They had around 70 sites across Britain, and my task was to create a separate page for each one, including photographs and information. The photographs arrived in a large envelope, and I scanned them all myself before building the pages.

It is difficult to imagine now how manual this process was. There were no modern image libraries or automated publishing systems. Each page was created, checked and uploaded individually. Most of the websites we built were uploaded, initially to Demon, but later to Nildram-hosted space, for example:

garrick.com/HarknessSayers

but the National Trust site was hosted on its own domain.

Nildram at this time was only a father-and-son setup in a house, and one hilarious moment occurred when I called them up to fix an issue on their server.

“Fine!” was the reply. “But we’re watching football, so we need to wheel the servers into the lounge, so we can eat pizza, drink beer and watch the football while we fix it.”

They fixed it pretty quickly.

We also started using tools such as WebTrends to understand how people were using websites. Analytics were still very basic compared with today, but it was the beginning of the idea that websites were not just things you built and left alone; they were things you could measure and improve.

This was also the period when the Web started moving beyond being just an information service and into something commercial. I remember my first attempt at making an online payment. It was not PayPal, which did not exist yet, but some early online banking or payment system. The transaction took around an hour and eventually failed. At the time it felt like a glimpse of the future, even if the future was not quite ready yet.

Another vivid memory from this period was watching NASA’s Pathfinder mission land on Mars in 1997. Because I was trusted to work independently, I was able to watch the online feed of the landing while continuing with my work. It was an extraordinary moment: sitting at my desk, connected through this new global network, watching a spacecraft arrive on another planet.

Towards the end of this period I also worked on the UK Online Movie Database. This was another sign that the Web was beginning to move beyond simple company brochure sites and towards searchable collections of information. The idea that people could use the Internet to access large databases of information was still relatively new, but it was clearly where things were heading.

The database backend was developed by Virtual Internet (VI) in their London offices, using a Linux-Debian hosted Oracle database and connecting software developed in Borland Delphi C++, later, Java. I remember visiting their office, and being allowed to peep through a door at the Java developers, sitting in darkness, listening to very loud metal music. Our web front end sat on early Microsoft Internet Information Servers (IIS) and used some tricky code developed by Nildram to display movie showreels of about 5 minutes in length and the option of a 4-inch wide clip at 10 Mb or a 2-inch wide clip at 5Mb, pushing the boundaries of what could be done before streaming.

VI’s Java developers inspired me to try my hand at Java. I tried Sun’s own Java Nature Trail, but got lost! So I bought Laura Lemay’s ‘Learn Java in 21 Days,’ which indeed I managed, but Day 12 took 12 hours and hurt my head like hell.

VI and Nildram later became very big companies, with VI contributing a great deal in discussions with an organising body of the Web called the W3C.

The disparity between what sales people imagined and what could actually be done was highlighted when Micky, one of the directors, told me to be in the office at 7 am, when he called up somebody in the Microsoft labs, passed the receiver to me and whispered:
“Work out with them how to activate a printer remotely using voice only.”
The lab technician simply said:
“How did you get this number? This isn’t even public knowledge yet!”
How Micky got the number, I never found out, but he was a salesman and very charming over the telephone when he wanted to be.

Another amusing incident happened on my birthday. Two of the female staff members vanished into a tiny kitchenette and didn’t come out for hours. I heard chatter about a stripogram sometimes from male staff members, so I became convinced they had booked one. But as the afternoon hours ticked by, I became increasingly anxious, wondering what sort of stripogram was I in for? When the ladies emerged, they were carrying a huge birthday cake they had baked themselves!

Sadly, imdb.com soon filled the space our own Online Movie Database was attempting to fill, but much more efficiently, if less colourfully.

Looking back, this was the period when the Web stopped being something I experimented with in my spare time and became the foundation of my career. I had entered the industry almost by accident, but I was now part of a rapidly changing world where every project seemed to involve discovering something for the first time.

Part 4

You can always find a full index of all the articles on the page under About me in the main menu.

Web Stories

In 1997, interest in virtual worlds really took off. These three stories of mine illustrate this:

In America, an art gallery, concerned about the effect light and visitors’ hands were having on their exhibitions, chose to close for one day each month and instead offer time steering a dirigible (in this case, a miniature airship equipped with a mini camera connected to the web) around the galleries. I’m sure there would have been plenty of crashes, but I never found out, because I tried to book a slot but never succeeded.

I also took part in a Virtual Reality art installation at the ICA gallery in London. We were issued with very heavy headsets and let loose in a large space, equipped with guide ropes around a course and a few very short flights of steps. While on the headset you saw fields, rivers and butterflies crudely displayed, attendants constantly grabbed your elbows and steered you away from other visitors or the walls!

I also signed up for a free virtual world, precursor to Second World, on the Web. It was simple, but you could ‘buy’ a house and find a place of work or recreation. I remember sitting on my sofa in my virtual house and watching numerous ‘stray’ avatars enter, sit down for a few moments, and then leave! These were avatars of account holders who had unsubscribed, leaving their avatars to wander randomly forever! I found it slightly unsettling!