The conference continues from day 1.

distributed systems

By Vaidehi Joshi. Everything is a distributed system. As Leslie Lamport wrote, “A distributed system is one in which the failure of a computer you didn’t even know existed can render your own computer unusable.”

A system will have faults. Some may be latent. Then an active fault turns into an error. If the error isn’t handled, it becomes a failure.

A few different ways distributed systems can fail. A reply can be slow. Or missing. Or incorrect. Or inconsistent. In a somewhat vague high level way we circle around discussing CAP without ever mentioning it. I don’t think we need to get into the nuts and bolts, but it’s a good framework to at least discuss what can go wrong.

Key lesson, how a system works isn’t as important as how it fails. Often we like to talk about all the things that happen when a system works. What happens when it doesn’t?

tortoise and hare

By Erica Gomez. How quickly should we build software? A topic near and dear to my heart. We start with a discussion of how airplanes are built, slowly, over years, and then consider the 737 MAX and its problems. A little too much rapid deployment.

Read The Mythical Man Month. Learn from it, internalize it, then accept that everybody just ignores everything in it.

Introduce friction. I quite like this idea. It’s considered best practice to make everything a smooth automated process. But this can lead to high speed disaster as well. A bit of friction can prevent things from running away too quickly.

A lot of flawed systems take the approach that it can be fixed in software. Software is often the last component built, and it’s the most flexible. But this makes it extremely fragile as well.

files

By Dan Lu. Files are hard. Somebody on the internet said they were easy, but they’re wrong.

We began by writing some short code to update a file. Except it’s error prone in the event of a crash. So we create a log. Except there’s a bug. And we add fsync. And that’s not enough. It turns out to be pretty difficult to correctly update a file using the posix API, and even the experts get it wrong.

Also, the file system lies. There can be an error writing to disk, which has as much as a 33% chance of being handled. Journaling file systems have options, which are documented, which is to say if you’re the person that wrote this terribly complicated code, there may be a mailing list post from years ago to remind you of how it should work.

Also, disks don’t work. They have error rates in data sheets. They have error rates measured in the wild. They are not the same. SSDs have a lot of error correction. That’s to make them work, not to make them safe.

One takeaway is that large cloud data providers build incredibly complex systems to store data with custom kernels running optimized file systems on exhaustively tested drives in order to prevent data loss. Consumers, however, also have tons of data in aggregate, but do not benefit from this work, because when consumers lose data, it’s consumers harmed by the loss. And so application developers have little incentive to fix their code, let alone drive manufacturers.

modularity

By Kamal Marhubi. We began our story with Barbara Liskov, who won the Turing award for data abstraction, and gave a speech mentioning a number of other papers. These were from the era of the software crisis, when it was difficult to write correct software.

Go to is considered harmful. That’s not really the original title, but people remember it. Also, it’s a pretty short article. Only one page really.

Another paper was the dataless programming language. We can write our code such that we can replace an array with a list simply by changing the declaration, and the compiler will generate the correct for iteration. This works for lists and arrays, but may not work for color pictures. That’s a tough problem.

Also, a paper by Parnas which is like a developer fortune cookie. One such note is that the connection between two pieces of code is not merely the physical structure of the interface and its types, but the assumptions being made.

To return to the present, we can’t just encapsulate everything or we end up with needless abstraction. But a useful technique is to leave a comment in the code noting which changes were more difficult than they should have been. If we return to this code, and have to make another such change, it may be time to redesign it. I thought this was useful. Experienced developers know where the pain points are in a codebase, but often this knowledge is hidden in the revision history where newcomers are unlikely to find it.

software history

By Hillel Wayne. Everybody knows SOLID and that it’s important, but why? Because somebody wrote a book, and then a lot of books, and said so. One wonders what other practices are basically the result of just so stories that we’ve forgotten the origin of.

How about the interview question about reversing a linked list? This will be an example which drives the process of asking questions, looking up sources, and finding out why.

Ask around, and people will either say it’s because it tests CS data structure knowledge, or it tests that you’re a quick thinker. Even though these two explanations are somewhat contradictory. But everyone agrees it’s traditional. Candidates are even asked to perform this task in python.

The origin of the question seems to be around 1980, when a sharp increase in programming jobs opened up. We can examine Usenet posts from the time to learn about these jobs. It seems C was a popular language. But a university degree was not a requirement.

So a new theory emerges. Asking about linked lists is a proxy for asking, do you know C? In other languages of the time, these would be provided, but in C you needed to roll your own. Thus manipulating a list would be expected to correlate with experience. But it was neither a test of CS knowledge nor quick thinking.

Then we can trace the question forward again, to see how it becomes engrained in the interview pantheon. Interview best practices use the question as an example. New companies read old blog posts, copy the question, etc. And now we have candidates for a python or ruby position answering a question originally intended to specifically identify C experience because we’ve forgotten why.

There are many such practices and questions that can be asked, and the answers are available if only we seek them out. sync; sync; sync.

odyssey

By Hillary Anderson. Common question is how does one learn programming, or a related skill? It can be something of a roundabout odyssey. In this case, as a political campaign data analyst seeking to learn more about data modeling, but generally applicable too.

Here are a few ways to learn. Boot camp. Can learn a lot, but also possibly high pressure. Self study. Learn at your own pace, but difficult to know if you’re skipping something important. On the job. A great way to pick up an adjacent skill. You work with a particular tool or data from colleagues, and now learn to make your own.

Here are three things that help learning. Community. Working on sample problems in a group can be encouraging. And people will offer suggestions, which even if unhelpful today, expand one’s knowledge. Accessibility. Working on a program that one understands and is motivated to solve. Have a reason to learn. Teaching. You don’t know it until you teach it. Interesting example about making a lesson tangible. Assign students to virtual results for a SQL query, then show the class a query, and have them stand up. This draws out the lesson a little longer, and engages the class, versus a thirty second slide easily glossed over.

statistics

By John Feminella. We can relay information by telling a story or by presenting statistics. Both have their uses. Mostly though we’re going to talk about some introductory statistics lessons.

The average, or mean, is where we start, but it’s not great. It doesn’t capture variance, or uncertainty. So maybe we can calculate a normal distribution.

That works, except on Black Friday. Black Friday traffic levels are not normal. So a reminder to consider the story, and not just the statistics.

The Datasaurus Dozen.

Understand your base rate. Very rare events require a very accurate test to correctly identify.

adios

See you next year.