Embedding Documentation and References

Posted on October 29, 2015 by R.P. Churchill

Documentation has traditionally been maintained separately from but in parallel to software systems as they’ve been implemented and maintained. Keeping the documentation in sync with an evolving system is a well-known challenge.

Documentation itself comes in many forms and even if the external documents are not maintained with sufficient, shall we say… enthusiasm, there are many ways to mitigate the pain of understanding older code and data. They include, among others:

  • solid naming conventions. Hungarian notation was one version of this that is now considered somewhat passé, but there are other aspects of naming things that still apply.
  • comments embedded in code, especially if they include dates, names, and reasons
  • comments added to source code control activities
  • minutes of configuration management board meetings
  • engineering notes
  • partial documentation. For example, the description of configuration files might be updated when changes are made because the users need it, even if the formal documentation is never touched.
  • e-mails
  • automated analysis, diagramming, and documentation tools
  • example data artifacts

It might be difficult to embed information about the code into the working system (beyond the idea that some languages are designed to be somewhat self-documenting) but any part of a system that is modular can certainly have its metadata embedded.

The screenshot below was from a tool I created to document and calculate the parameters and starting values for the thermo-hydraulic models I wrote for nuclear power plant simulators. The entries in the upper right corner are to record the document numbers where the supporting information came from. The plant I primarily modeled sent close to 10,000 copied documents to our office, each of which was assigned an index number. Additional documents created as letters, emails, memoranda, technical updates, and the like were added to the library over the course of the project.

I was thinking about this because of a recent project I worked on had to regularly incorporate data sets from many similar sources in several different formats. The organizer in charge of acquiring, formatting, and collating this information every year had to maintain the contact information for every co-worker who provided the information. If that person was replaced during the year the organizer had to figure out who the replacement was. Those responsible for providing the information didn’t always remember what had been provided previously, how it was done, or what it meant. The content, format, and labeling of the data might also have changed in the interim.

In theory the organizer could have gotten the providers to document all of those procedures and labels and forward that metadata along with the actual input data. It could then have been maintained in a fairly neat way, apart from how the inputs were folded into the final product. The metadata information could have been printed and placed in a notebook, organized into a single document, or saved in some other way. If this information were saved each year, along with the contact information of the providers, both the organizer and providers would have maintained a much better memory of how things should be done from year to year. As it was it was up to the organizer to do a lot of cajoling and hand-processing and recreating the crime every year to get the needed data.

I encountered this organizer when assigned to a project meant to calculate the number of staff needed to support the yearly volume of activities carried out at each of a large number of locations. The project’s purpose was to automate the system to standardize the incorporation of the input data, identify errors in the content and format of the data, perform the calculations in a more reliable and consistent way, and make the results available interactively to a wider range of people.

We therefore had the opportunity to define any data structure we wanted. It occurred to me that we could embed the metadata for all of the formal input artifacts and procedures into the data structure (and user interface) itself. We could also make the program keep track of which data inputs were updated and when, which at the very least could serve as an embedded to-do list that ensured everything got done.

Input data usually came in the form of individual spreadsheet pages which would then be folded into the target spreadsheet as new tabs. The data could also come as comma-separated variable (.CSV) files, as database files, or as raw text. As the automated system was expanded, it could have been connected directly to the live databases from which the contacts pulled the information they sent to the organizer.

The activity information received was usually formatted as tables. The columns usually described activities while the rows were usually associated with locations. At least one of the columns had to contain information about the location so the data could be collated on that basis. There were some exceptions, but let me describe what a good metadata description might have included for the standard case where the information came formatted as a standard table:

Contact Name: name
Contact Phone Number: phone
Contact E-mail: e-mail
Contact Office Location: office
Data Artifact Type: .CSV file, Excel File, etc.
Date Received: date
Time Received: time
System Data Retrieved From: system name
First Header Row: r1
Last Header Row: rn
First Column: c1
Last Column: cn
Column containing Location Code: col
Column 1 Header Line 1: header text 1
…
Column 1 Header Line n: header text n
Column 1 Meaning: description
Column 1 Units: units
…
…
Column n Header Line 1: header text 1
…
Column n Header Line n: header text n
Column n Meaning: description
Column n Units: units
Link to Example Artifact: file location and name (or web url)
Description of How Data Was Acquired: Provider’s account of how the data artifact was produced

Now we have a means of ensuring that the organizer asks for all of the artifacts needed. The organizer knows who in the organization to contact. The request for the information could be automated. It would be nothing to have the system embed this information in an e-mail saying, “Hey, I’m <so and so>, the organizer for <the whatever>, and here is the data you sent me last year, its meaning and its format, and how you produced the data the last time. Please send me the updated data by <whenever>, or contact me if there are any difficulties.”

If the implementers were really ambitious they could use the metadata information to test the content and format of the received artifacts automatically, particularly if the metadata definition incorporated information about what constituted valid values for each type of information. The information would also be available within the system’s UI, say, via a context menu that comes up when the user right-clicks on a relevant data item.

If the system were enhanced to automatically link to an external database to pull data directly, the metadata could be expanded to include the location and name of the relevant database, credentials needed to log into it, and the text of the query or queries needed to generate the desired data. Moreover, the automated process could be automatically run at intervals to ensure that the retrieved information is still accessible and valid. If the automatic access fails, the organizer and provider can be automatically notified so they can figure out what changed and make the necessary updates.

I noted above that any modular part of a system could have metadata associated with it, and this can include modular code. The system in question included modular scripts that performed calculations, and allowing the user to see and maintain both the scripts and the related meta-documentation would have gone a long way to ensure that the documentation was kept up as well as the “code”. There were also cases where different sources of data were used or different calculations were used for activities at one or more locations. Metadata could be stored about exceptions and overrides as well.

Information about who defined or modified each modular element can be included as well, along with the date and time of those events, rationale and descriptions, permissions from approving authorities, links to configuration management meeting minutes, and so on. This metadata could then be queried and filtered to augment similar records that may or may not be kept elsewhere.

The point is that there are a lot of ways to make a system easier to maintain. Some parts are difficult. If you change the underlying code then you should do the work of updating the documentation where needed. This requires diligence and good management. If you can build procedures to automate the documentation and configuration management of the modular parts of the system, then in this day and age, where the constraints are less on system resources like speed and memory than on time, expertise, and management of complexity, it’s almost criminal not to do so.

Posted in Software | Tagged , , | Leave a comment

It Isn’t Always You (It Usually Is… But It Isn’t Always)

Posted on October 28, 2015 by R.P. Churchill

I’ve spent a lot of hours and years grinding out code. This means I’ve had to solve a lot of problems, many of my own making. Seriously, almost every coding problem is the local programmer’s fault and can be fixed by that programmer. If that person can’t fix it, find someone who can, even if they’re only going to Google StackOverflow. Or, as Jeff Duntemann would say,

“There is in old saying to the effect that, after you spend hours banging
your head against a bug, someone completely foreign to the situation will
walk into your office and immediately spot the flaw. It’s true, and
something you should use often. If necessary, lure them in with food.”

That said, if elbow grease and contemplation and help and Google and dishes full of sweets don’t get you anywhere, you might just have a problem with the tool. I’ve run into four that I know of.

1. The first data archiving system I wrote, for a mill in Kansas City, made use of variant records, also called free unions. The system stored the heating history of every billet processed by the furnace, and these files were appended every 50 seconds. All of the records had a fixed part and a variant part. The variant part was needed because the archive files were based on a fixed record size and the header information (the first entries in the file that described the billet’s ID, dimensions, chemistry, and so on) were much larger than the information that needed to be stored during heating. If all records were going to be of the larger size then the files would be filled with a lot of empty space, which kind of hurts my soul. Oh yeah, and there wasn’t that kind of disk space to spare. The solution was to break the header up into two or three separate records so each would be the same size as the heating records. I wrote the code to write the records into the files and everything worked famously.

Until I went to read the files and display the results.

When I read the records back out the information in the variant part of each record was garbled. Looking at it bytewise I saw that the data had been shifted two bytes in the variant part of the records. I struggled with this for two solid days. The target system was written in FORTRAN for a DEC VAX machine running VMS. I was developing the code in parallel in Microsoft FORTRAN PowerStation 32 (on the PC, I think in Windows NT) so I could debug more efficiently and work at the hotel in the evenings. After a few calls to Microsoft they recorded the problem as a bug (the compiler was aligning one of the items on a four-byte boundary when it should have aligned it on a two-byte boundary) and told me that it worked just fine if I changed the order of declarations in the variant part, so the first data member was of a different type that the compiler would align correctly. I did so, and the fix worked.

I tried it both ways on the VAX, and both worked. Its compiler did not insert the extra two bytes and I might not have needed to spend so much time on it.

2. I was in a good flow banging out model code in Borland C++ Builder for a mill in Memphis when my ongoing testing showed that one of the statements I wrote wasn’t executing. I went back several times to step through the code in the debugger and when stepping through one line I saw that the calculated value did not change; the code was having no effect. I looked at it over and over and kept coming up with nothing. I finally decided to insert a dummy statement just ahead of the offending line. I set the value of a global variable so the compiler wouldn’t optimize away a similar operation on a local one. Boom. It worked. The calculation was now carried out and that result was generated as expected.

That one hurt my soul a little bit, too, but I could live with it.

3. The next one also came up on that project in Memphis, but it was different. One of the other vendors wanted me to send him files via FTP, so I used the FTP component provided by C++ Builder. It worked just fine, but every morning I’d come in to find that my system was locked up. I eventually figured out that there was a memory leak in the FTP component, about which I could do nothing. The other vendor accepted my proposal to read and write files on a mapped drive instead, which solved the problem.

4. The last one I found was in SLX when I first got to my job at Regal Decision Systems, in Baltimore. I was writing a tool to simulate the operations in medical offices when one of my runs crashed in spectacular fashion. I was able to identify the offending line in the debugger but everything looked in order to me. I contacted the gentleman who maintains the SLX code base and sent him the code and data I was running. He was able to reproduce the error, which was lucky enough — or is that unlucky enough? — to break a piece of SLX’s code in a way that hadn’t been done before. He said that particular piece of code must have run billions of times without breaking but did not offer specifics about what went wrong.

The bottom line is that if you code long enough you will almost inevitably encounter issues with the tools you’re using. Remember that the people who create the tools are just like you, though possibly with more education and experience and with better management. Sometimes you can find a workaround on your own and sometimes you can’t. In the latter cases I hope you’ll be able to get the help you need.

Posted in Software | Tagged | 1 Comment

When I Knew I Was “Home”

Posted on October 28, 2015 by R.P. Churchill

When I was very young my family sometimes took the train from DC to New York to visit my grandparents. I still remember the swaying and rumbling of the cars and the smell of oil and a certain kind of grime. It always seemed to be raining. The tracks were naturally laid through the industrial backyards of New Jersey and when you weren’t looking at mountains of used tires headed for disposal or recycling you tended to see sights like this:

Vent flares gave the scenes a hint of animation — and menace. I didn’t know what was going on out there in the dark but it seemed adult and happening, and yet somehow very far away, as if I could never grow up and be a part of it.

Years later I’d gone through college and the military and finally found myself in Shawinigan, Quebec, walking through a paper mill, tracing a pulpmaking process across the sheaf of drawings I’d brought. The last drawing showed the steam regenerator. It was eighty or a hundred feet high and I found it outside in a gap between buildings. I was on an elevated walkway about halfway up and just stopped to marvel for a bit. Eventually I looked around and saw a single light fixture, a flood lamp mounted in a protective cage something like this:

Then it hit me. The whole paper mill wasn’t too different than the oil refineries I’d seen way back when. There were hums and lights and columns and pipes and jets of steam and big things happening. The smells weren’t the same. The weather was nicer. It was daylight.

But I knew I was finally where I was supposed to be.

These days I’m just as happy in a customer’s office or diagramming away on a whiteboard as I would be at any industrial site. As long as I’m figuring out how to make a process better, and as long as I get to work it out with smart people, I’ll still be right at home.

Posted in Engineering | Tagged | Leave a comment

A *LOT* of Ways to Test Software and Systems

Posted on October 22, 2015 by R.P. Churchill

Traditional Development Model (V-Model)

Agile Testing Quadrants, as proposed by Brian Marick

Having developed software for systems large and small, as well as performed every kind of software testing services known to man – I’ve been involved with most kinds of software testing. Here are the major details from my experience.

The traditional way of defining and conducting tests is shown in the V-Model, and the more recent Agile, Four-Quadrant concept of testing is shown below. While trying to classify the methods I’ve employed over the years I see it is difficult to place them neatly to one “box” or another in either model. The V-Model places integration testing halfway through the process while the Agile Quadrant model seems to place it near the “beginning”. However, the two models are really different temporally; the Agile Quadrant model suggests time more loosely than does the V-Model, but in the real world most tests are conducted cyclically anyway. Good developers didn’t need to learn Agile and Scrum formally to know enough to test, demonstrate, and seek feedback early and often. Therefore I will group my experiences in my own way.

Testing Local Code

This is all about getting your own code to work. As much as possible you want to not have to worry about interactions with external hardware or software. This is the most granular type of test. New capabilities that automate a lot of this process make it much simpler to find problems in code long thought to be stable as other parts of the code are added or modified.

  • I’ve used JUnit in Java development. It enables Test-Driven Development by supporting the creation of tests prior to writing the code. They fail before the code is written and then, hopefully, pass after the code is written. These independent test functions can be as simple or complex as necessary, and streamline the kind of work that used to get done in other ways. They are generally intended to be small, numerous, and automated. These are mostly used to test local code but can be used to support integration tests as well.
  • I learned a little about Mockito when I was qualifying for my CSD. This clever bit of subterfuge bypasses the behavior of the function actually called and produces the developer’s defined behavior (the mock) in its place. This allows code to run in a standalone mode when you don’t have access to connected systems, third-party libraries, or other functionality. I’ve generally thought about this kind of spoofing to be a higher form of testing, like functional or integration testing, but in this instance the intent is to support testing of the local code, not the interactions. One can also spoof these behaviors in less elegant ways. The real code can be commented out and the desired response can be dummied in, but then the developer has to remember to change the code before release. Conditional debugging statements can support the same behavior in a slightly more elegant way. Using mocks has the virtue of leaving what is hoped to be the final code in place in its final form and without the need for changing any switches.
  • A lot of my development has been perfected through classic interactive debugging tools, beginning with Turbo Pascal (Version 4.0, I think). Most subsequent programming environments have supported this. That said, there are cases where this method cannot be used effectively, and more primitive methods have to be used instead. As Jeff Duntemann once observed, “An ounce of analysis is worth a pound of debugging.”
  • In more primitive situations I’ve resorted to the classic method of embedding write statements in the code. Sometimes I’ve only needed to write out critical values in the area of the problem, but other times I’ve had to write out much more detailed information, including essentially a full stack trace. This method is also helpful in real-time or other interactive situations that would not operate with the potential delays inherent in interactive debugging.
  • In rare cases I’ve used sound to provide runtime information. Power-On Self-Tests (POST Tests) for computer boards often do this.
  • In many cases you can tell what’s going wrong simply by looking at the output, of which there are too many kinds to list.
  • I’ve often developed small sections of code in isolation, and then integrated the whole into the larger project. This was always good for 3D graphics, communications widgets, matrix calculations, and automatically generated code. It was also useful for independently graphing the outputs of functions for material properties and random distributions. This may also be thought of as component testing.
  • I wrote a simulation framework that allowed the developer to display any desired variable after any time step. In some ways the system behaved like a giant set of watch statements in an interactive debugger. The power plant simulators I worked on provided the capability that initially inspired my project. The tool I wrote also contained a number of graphic elements like multi-trace scrolling graphs, animated bar graphs, and other moving plots.
  • The simulation framework allowed me to vary the states of external variables that were referenced by the model. This could be done in simple or complex ways. The simulation could be frozen and such a variable could be edited by hand, or the interface variable could be made to change automatically in a prescribed way (e.g., sine wave, random walk, programmed or random step change, and so on).

First version of my interactive continuous simulation testing tool

Integration Testing

Once you get all the pieces working you see if they can play nice together. This is applicable to testing systems that interact with the outside world, with other computer systems, with other software processes on the same machine, or with specialized services like databases.

  • Many of the systems I worked on integrated multiple software processes on a single machine or a group of tightly coupled machines. The processes often communicated through a single block of shared memory. Testing these interactions meant ensuring they were all using the same definition of the shared memory space, ensuring that certain values were locked and unlocked as needed, that race conditions were avoided, and so on.
  • In other cases my systems had to communicate with external computers. This was usually though high-level protocols like TCP/IP, FTP, DecMesageQ, and writing files to commonly-mapped disks (hard drives or RAM disks). The messages, protocols, addresses, and permissions all had to be verified. I was never able to do standalone testing on the DEC machines we used in steel mills but for PC-based systems I created my own test program that replicated the behavior of all external systems either exactly or nearly so. That way I could run and test my own programs unchanged and have them respond to every possible change in the external systems. I even made the external test program respond to the commands to change its parameters.
  • Plantwide systems started to integrate database access as a means of interfacing higher-level systems. Rather than wait for a semaphore or other message the systems would actively poll a database on an interfacing system. The code would execute defined SQL scripts we developed to perform the same functions our earlier interfaces had, complete with locking and re-trying loops.
  • Systems that interfaced closely with external hardware needed to be physically connected to it. I tested systems, programs, and drivers that interfaced to a broad assortment of hardware devices though RS-232 and RS-482 serial connections. Some of these involved publicly published general protocols while others were specific to individual devices. Some involved TCP/IP messages. In the case of a system meant to control induction melting furnaces the core system had to interact with a number of serial devices, some of which were themselves in communication with multiple sensors and controllers.

Level 2 System Architecture Showing External Systems Spoofed by Test Program
UI for External Interface Test Program

User Testing

This type of testing involves examining the way users interact with the system. This can be through hardware or software, though my concern was almost always with the software.

  • I learned about Selenium while qualifying for my CSD. That seemed more robust than previous automated UI testing methods that involved macros that embedded mouse movements and actions.
  • Most UI testing I’ve been involved with has been manual. That said, sometimes the users have gone off to do their own thing and sent back comments, while in other cases I sat with the users or managers and received their feedback directly. In many cases I was able to make the desired changes on the spot.

Business Acceptance Testing

These are higher-level tests meant to address business behaviors the customer cares about. They are conducted at the level of a user story, a business function, or some other “chunk” of functionality.

  • As part of a team that performed an Independent VV&A on a major analytical system (a deterministic model meant to manage fleets of aircraft) I saw a very neat form of testing. In this case the architect of the system wrote the major functionality in Python, based on the developed requirements. The development team implemented the same two core modules, which read the same inputs and generated the same outputs, in a C-based language to be served over the internet. The clever part was that the two core modules written by each of the players wrote out debug files that described every element they took as input and every element they generated as output. External comparator programs then automatically compared the two debug files for each of the two major modules and cataloged the exact differences between them. The number of elements compared for some of the elements was over 100,000. This testing could not prove that either system was correct per se, but if two developers could interpret the same set of requirements the same way, that could be a good indication that they were on the right track.
  • I learned about Cucumber as I was qualifying for my CSD. That leverages specific statements in a list of business requirements that can be tested against.
  • All of the control systems I wrote had to ensure that the furnace produced steel that would be rolled properly in the rolling mill. It also had to ensure that the temperatures predicted by the model and controlled to by the model had to match what could be physically measured. A system could theoretically calculate any number for steel temperature as long as control to that number resulted in steel that could be rolled properly. The number could be 37, 2140, or negative one million, but that would be very confusing. Therefore the system not only had to produce steel that could be rolled properly with a clean surface (that acceptance criteria was governed by a different part of the control system), the predicted surface temperature had to match the temperatures recorded by an infrared pyrometer as each piece of steel was discharged. This was more correct from an operational point of view and gave the users a much greater sense of confidence.
  • Simulations could be verified in a couple of different ways. One was to see if a set of inputs will reproduce a known set of outputs, which come in many forms. Once it is agreed that the system matches known outputs the system is tested to see if novel sets of inputs result in outputs that make logical sense.
  • In some cases the veracity and applicability of a tested system’s outputs can only be evaluated by subject matter experts (SMEs). In those cases you get them to work with the system and its outputs and have them attest to their acceptability in writing.
  • The training simulators I helped build were contractually obligated to match the real plant’s operating indicators within 1% at steady state and to within 10% in transients. Verification of this behavior could only be carried out by experienced plant personnel interacting with the simulator as it ran.
  • I worked on a project that first had to duplicate the results of the existing system exactly, even though some of the existing outputs were found to be in error. I was able to use the new system’s reporting functionality to compare roughly a 100,000 values from each system, which indicated that we understood the original model perfectly. We were then free to correct the errors in the original model.

Technical, Performance, and Reliability Tests

These tests address the behavior of the system as it is installed and examine qualities like performance, load, and security.

  • I was directed to install a system on a plant computer that also hosted a database server. My system threw an alarm whenever it was not able to run within a certain period of time, and the alarms started showing up for a few minutes at a time two or three times a day. My system’s event logs told me when the events happened, but couldn’t tell me why. I figured out that the Performance Monitor program in Windows can archive the performance of a wide range of system resources and after a day or so I was able to figure out that a particular function of the database’s operation consumed so much CPU and disk I/O that little else was able to run. I ended up installing my suite of programs on a different computer that wasn’t hosting any processes that consumed too many resources.
  • The pulping systems I supported were contractually required to be in operation 95% of the time. I helped create a different system that was allowed a specified maximum amount of downtime.
  • Other systems I installed were expected to run more or less forever, though it was a good idea to reboot them during down periods. The systems I wrote for the steel industry had limited hard drive space that was easy to fill up with historical archive files. My solution was to add a process which periodically erased any such files above a certain age. The plant personnel were in charge of saving the relevant data to a more permanent location before it was lost.

The bottom line is that systems have to meet a wide range of logical and usability requirements and a wide range of test procedures are needed to examine each. The more tools there are in the tester’s toolbox the more quickly, completely, and accurately they’re going to be able to ensure they’ve got it right.

Posted in Engineering, Software | Tagged , , | Leave a comment

A friend wanted to work on a “great” project

Posted on October 22, 2015 by R.P. Churchill

A fellow engineer once commented that our work seemed mundane. He wanted to do something big or spectacular or “great”, like the moon landings or Hoover Dam. And it wasn’t just that he wanted it to be great, he wanted to work on something that was of historical importance. “What’s our generation’s big project?” he asked.

That’s all well and good, and our work wasn’t the kind that got written up even in popular technical magazines, but I think that maybe our work was great. We are living through one of the most transformative technology revolutions in history. The computing industry had been around for a while before I started but things have really hit critical mass since then. The growing power of computing and communications technology has improved things and made possible things that simply could not have been done before.

Building the hardware and software for full-scope nuclear power plant simulators was a complicated undertaking that required lot of expensive equipment and a large team of people. Computers of that time were only just becoming able handle the I/O needed to run the forest of control panels, calculate the status of the valves and switches and pumps, and simulate the electrical and fluid systems in a plant smoothly.

Flow vs. pressure calculations are a square root function and the generation of simulators we worked on were powerful enough to compute those in quantity, where earlier computers had to approximate by using two linear functions. That limited the accuracy that could be achieved but was sufficient for studying normal operations. Operations involving major transients had discontinuities in detail but were at least reasonable in character.

Simulations from still farther back were built on analog computers. They were composed of physical circuit links whose resistors, capacitors, and other items which made each link behave like its analogous fluid or mechanical connection. One of the older engineers I worked with described zero-slice and one-slice functions but I could not find references to those. My understanding is that they were low-pass and high-pass filters that had the effect of performing logical truncation.

Full-scope simulators of the time took several years to build. Technology was changing so quickly that PCs were capable of duplicating many of their functions by the time the simulators got delivered. I was surprised that the PCs I owned were good enough to develop and test complicated models in real-time while also generating pretty graphics. In subsequent years the big Gould/Encore SEL 32/8000-9000 series minicomputers and Sun workstations were replaced by PCs entirely.

I always thought that working on those systems was one of the most fun things I ever did, but others were not as impressed. Indeed, I’ve been able to ride that ever-increasing computing power to do all kinds of newer and better and faster and cheaper things. My engineer friend stayed in the nuclear industry, during which time its safety, reliability, and productivity have improved markedly.

That said, an engineer who actually worked on the Apollo program found his way to Westinghouse when I was there, and he proved to be very talented. (I always found it interesting that so many of those engineers were so young.) I remember that he ran afoul of a consultant who had a habit of gumming things up on one of the projects, and the last time I saw him he was sitting with a different co-worker, describing his disagreements with the consultant while slamming out differential equations in anger. I don’t know whether I thought highly of him because he had worked on Apollo or just because we was a good engineer.

It may also be that fighter aircraft or iPhones or other very specific, highly visible projects will always seem sexier than grinding out analyses and training and work-a-day products. I guess it’s all in how you look at it. Maybe I’ll ask my friend if he feels differently now.

Posted in Engineering | Tagged | Leave a comment