Category: Debugging

How to build a PostgreSQL-backed website. Quickly.

Model-View-Controller using Ruby-on-Rails & PostgreSQL

So you want to build a website? Quickly! But able to grow to arbitrary size, if fortune smiles.

I’ve been working on just such a website, one which runs the manufacturing process for a world class optical switch manufacturer. They started in 2005 and are now the 2nd leading producer of optical switches. So zero to hero. They started with Ruby-on-Rails & are still running the original code. With a few hacks of course, many of which I built.

So, time for a talk on how to do this. The 2021 PostgreSQL Conference Webinars in the person of Lindsay Hooper was kind enough to extend an invite & I put something short together. I gave the talk earlier today as a webinar. Very happy with the way it came out; it provides a short introduction to how to get started on this, from the database perspective:

I show how to get started building a PostgreSQL-backed website using Ruby-on-Rails, what is meant by the Model-View-Controller architecture, why we want to use that, what tools you need to get started; how to work with the online tutorials; what kind of workflow to use, and which tasks to let Ruby-on-Rails handle versus which are better done by PostgreSQL.

The conference will be posting the video of the webinar in a few days. Meanwhile, I have exported the slides as a PDF up on slideshare.

And if you are wondering what the illo has to do with the talk, it’s a breakdown of the pieces of the Model-View-Controller architecture as it applies to Ruby-on-Rails/PostgreSQL. If you are going to do this, it is the map which is your world.

The Elephant Versus the Bug

Debugging with PostgreSQL -- the Elephant takes on the Bug

Depending on the project, debugging can take 50 to 90% of development time. But it usually gets less than 10% of the press. PostgreSQL has great tools for debugging, but they are most effective when deployed as part of an overall strategy.

Ten days from now I’m doing a webinar on how best to do this. That’s 1pm, July 29th, 2020. You can register here.

It’s a fun talk: I focus on the key ideas, give a few good examples, have a quick quiz “There are three bugs on this slide: you have the next two slides to find them!” and then finish with some of my favorite debugging references: selected as particularly readable & practical.

[Update: the talk was given as planned. Thanks to Lindsay Hooper for great support! Video is online. And the audience of invisible Zoomers upped the number of bugs on the quiz slide from three to five!]

Last time I did this talk, my favorite piece of feedback was from Bruce Momjian, a founder of PostgreSQL and a member of the core team: Bruce said there was stuff he knew but hadn’t heard put into words, and stuff he just hadn’t seen yet.

And that’s how I hope this will be for you!

We will look at strategies for debugging PostgreSQL: how to find bugs, how to fix them, and how to keep them from happening in the first place.

We’ll look at root causes, technical tricks, and scientific strategies, and why — even if you can’t always write perfect code — it is usually a good idea to try.

We’ll hear from Bjarne Stroustrup, Sherlock Holmes, Kernighan and Ritchie, Pogo, & the experts of the PostgreSQL community.

Goal: less time debugging, more time building great tools and apps that stay up & get the job done.

If you can’t wait or just can’t make it, I have the latest version here: PDF, KeyNote, and PowerPoint, name your poison.

Update

Talk went off smoothly, about 65 or 70 attendees. The invisible Zoomers found two more bugs on the “quiz” slide! The event organizer, Lindsay Hooper of the PostgreSQL conference, did a great job setting every thing up, watching over a dry run, and giving feedback during. And she recorded and edited the webinar, so if you are interested in see the video “live”, herewith.

Debugging with PostgreSQL – Sample code

My talk last week at FOSSCon, “Debugging with PostgreSQL: A Strategic Approach” went well. Lots of energy in the room. Good audience.

Bruce Momjian, one of the founders of PostgreSQL, was in the audience & said afterwords (roughly):  “that’s what I’ve been thinking for years; good to hear it spelled out in words”. I got that from a number of other programmers in the audience as well. Much pleased.

Bruce went on to ask I propose the talk for the 2020 World PostgreSQL Conference, which I shall.

I thought it might be helpful to write some of the code examples up in a complete script, so any one who wishes can run and/or hack. I found a few problems and infelicities myself while doing this. Further suggestions very welcome!

Warning: here there be code.

To run the code (assuming you have PostgreSQL 11 installed and call the sample “sample_all.sql”):

psql -U postgres -d postgres -f sample_all.sql > sample_all.out 2>&1

Since it can be tricky to cut-and-paste from a web page, I have uploaded the raw code as “sample_all.txt” (you can’t upload files with an SQL extension for security reasons). For completeness, here are the slides themselves as PDF.

The code is careful to create a sample database, build & test stuff, and then remove the whole thing as if nothing had happened. If you don’t like doing this sort of thing from the postgres user (don’t blame you) create a user with createdb privileges & use that to run this.

Sample Code

/*
	John Ashmead 
		sample_all.sql:  samples as used in my talk "Debugging with PostgreSQL"
		FOSSCon 8/17/2019

	Sample_all.sql is a complete code sample:

		it builds a sample database called sample with a user sample

		then creates a few types, 
		a timestamp trigger function, 
		a table people, 
		and then a small function to set the social security number

	The goal was to provide illustrations for the talk of what I call "self-debugging code"
		1) many problems are trapped, as by type checking, before they can do any harm
		2) in other cases, you will get an exception
		3) and in the worst case, at least you will see what went in and what came out
		
	You can run this as user postgres database postgres.  You could run as any user with createdb,
	if you fix the clean section to go from "postgres" to that user.

	I normally run scripts using psql with "-v ON_ERROR_STOP=1" set on the command line, 
	which will cause psql to exit on the first error.
	
	But in this case you need to allow for errors in the test section. 

	Therefore an appropriate command line is: 
		"psql -U postgres -d postgres -f sample_all.sql > sample_all.out 2>&1"

	The comments are taken from points made in the talk,
	hence their perhaps slightly pedantic character.

	Any comments, my email is "john.ashmead@ashmeadsoftware.com".
*/

\qecho Build user and database
create user sample with password 'HighlySecret';

create database sample with owner = sample;

\c sample sample

set search_path to public;

/*
	Create generic timestamp function: timestamp_trg

	Provided the tables use the fields "updated_at" and "created_at" as timestamps,
		you do not need to rewrite this function on a per table basis.

	It is very useful to have timestamp fields on most tables, even if they are not specifically needed:
		1) knowing "when" something went wrong often takes you much of the way to figuring out "what" went wrong
		2) and using triggers takes the load off the development programmer
		
	I've been working a lot with Ruby-on-Rails which will create & update these fields for you.
	But if you rely on Ruby-on-Rails then you create a lot of traffic on the wire,
	and you can miss cases where the updates were done behind ruby's back,
	as by other scripts & tools.
*/
	
\qecho Create timestamp function
create or replace function public.timestamp_trg() returns trigger
    language plpgsql
    AS $
  begin
	/* assume we have updated_at and created_at fields on the table */
    if new.created_at is null
    then
      new.created_at = now();
    end if;
    new.updated_at = now();
    return new;
  end;
$;

/*
	My own experience has been that it is much better to use logical types, even for simple fields:
		1) it makes changing types much easier:  if three tables are using a social security number, 
		then you only have to change it in one spot
		2) it makes the field names almost self-documenting
		3) and you can include bits of validation, as here, when the field is used

	Obviously this, like any principle, can be carried to extremes.  
	This is, as Captain Barbossa might put it, a guideline rather than a rule.
*/
\qecho Create some types & then the people table

begin;

/*
	Everysooften you run into someone with a single character last name, as Kafka's "K",
	so we allow for that.  

	I prefer text to varchar or character.  Performance about the same (in some cases better) and 
	if you put a fixed length in, what happens when you have to add the last name of a king or queen
	where the name is basically the history of the monarchy?
*/
create domain public.lastname_t text check(length(value) > 0);
comment on domain public.lastname_t is 'holds last name.  Has to be at least one character long.';

create domain public.firstname_t text;
comment on domain public.firstname_t is 'holds first name.  Can be missing';

create domain public.middlename_t text;
comment on domain public.middlename_t is 'holds middle name or initial.  Can be missing';

create domain public.ssn_t text check(value similar to '\d{9}');
comment on domain public.ssn_t is 'holds social security number.  If present, must be 9 digits long.';

/*
	ok_t is self-documenting in the sense that true is good and false is bad.
	This seems obvious enough, but I have seen the reverse convention used.

	As an aside, it is better for maintenance to use positive tests, i.e. "if we_are_ok" 
	rather than negative ones "if not we_are_failed".  Slightly easier to read.
	Which is important when it is 2am and the code has to be working by 9am.

	Further, better to use "not null" whenever possible:  three valued logic is a great source of bugs.
*/
create domain public.ok_t boolean not null;
comment on domain public.ok_t is 'true for success; false for successness challenged';

-- PostgreSQL sequences are a joy!
create sequence public.people_id_seq start 1;

/*
	we are using the ruby convention that we should get the plurals right:  person/people rather than person/persons.
	The only place you see persons is in a police report:
		three persons of a suspicious character were espied leaving the premises in a rushed and furtive manner.
*/
create table public.people (
	id int primary key default nextval('people_id_seq'),
	lastname lastname_t not null,
	firstname firstname_t,
	middlename middlename_t,
	ssn ssn_t,
	updated_at timestamp with time zone default now(),
	created_at timestamp with time zone default now()
);

/*
	In this simple case the comments are, in all candor, redundant.

	But, if you comment everything, then tools like SchemaSpy can give you a nice report of everything in your database.

	And, it is a good habit to get into.
*/
comment on table public.people is 'list of people';
comment on column public.people.id is 'primary key of people table';
comment on column public.people.lastname is 'lastname of person -- mandatory';
comment on column public.people.firstname is 'firstname of person -- optional';
comment on column public.people.middlename is 'middlename of person -- optional';
comment on column public.people.ssn is 'social security number of person -- optional';
comment on column public.people.updated_at is 'last time this row was updated';
comment on column public.people.created_at is 'time this row was created';

-- A unique index on id will be created automagically, so don't bother. 

create index people_name_ix on public.people using btree(lastname, firstname, middlename);

create unique index people_ssn_uix on public.people using btree(ssn);

insert into public.people(lastname, firstname, middlename) values ('Programmer', 'J', 'Random');

select * from public.people order by id;	-- make sure we look OK

/*
	One useful trick is to put a begin at the top of a script & a rollback at the end,
		until you are confident that the script works OK.
	This can be done even for DDL -- i.e. create table -- an incredibly strong feature of PostgreSQL.
*/
	
-- rollback
commit;

-- create ssn_set
\qecho Create the social security function which served as the main example of self-documenting code

-- begin/commit not strictly needed, the create function is an atomic unit, but still a good habit
begin;

create or replace function public.ssn_set( 
	person_id0 public.people.id%type, 	-- makes certain the function & table types are lined up
	ssn0 public.people.ssn%type, 		-- lets us get in a bit of validation (against the ssn type) before we get started
	debug_flag0 boolean default false	-- this lets you turn on debugging at will, if there is a production problem
) 
returns ok_t as $
declare
	person_id1 people.id%type; -- more specific than int
	ssn1 people.ssn%type;	   -- could use ssn_t, but this is still more specific than a generic type
	row_count1 bigint;		   -- more check-y stuff
begin
	if debug_flag0 then
		/*
	   		notice the use of the function name in the message:
	   		always identify the source in an error message! this could be part of a thousand messages
		*/
		raise notice 'ssn_set(%, %)', person_id0, ssn0;	
	end if;

	select id into person_id1 from people where id = person_id0 limit 1; -- limit 1 is overkill
	if person_id1 is null then
		/*
			be as specific as possible in an error message
		*/
		raise exception 'ssn_set:  person_id0 % is not in people table', person_id0;
	end if;

	/*
		We have a unique index on the ssn, but we can issue a more precise error message if we check first.

		This also serves as a double-check if we set the table up incorrectly, unlikely for social security numbers,
		but can happen in general.
	*/
	select id into person_id1 from people 
		where ssn = ssn0 and id != person_id0;
	if person_id1 is not null then
		raise exception 'ssn_set:  ssn % is already in use by id %', ssn0, person_id1;
	end if;

	-- this whole function is really just an elaborate wrapper for this one line
	update people set ssn = ssn0 where id = person_id0;
	/*
		and now make absolutely sure that it worked
	*/
	get diagnostics row_count1 = row_count;
	if row_count1 != 1 then
		raise exception 'ssn_set:  unable to set ssn to % for person# %, rows affected = %', ssn0, person_id0, row_count1;
	end if;

	/*
		giving the exit values as well as entry values of key variables lets us trace
		the flow of gozintas and gozoutas without doing anything more than setting a debug flag
	*/
	if debug_flag0 then
		raise notice 'ssn_set: person %: ssn changed to %', person_id0, ssn0;
	end if;

	/*
		All previous returns were by "raise", this is our first "normal" return.
	*/
	return true;
end; $ language plpgsql;

commit;

/*
	and of course the obligatory red/green tests
	-- bracket the allowed value with three red tests, then verify it works
	-- then check for dups:  one red, one green
*/
\qecho Test the social security function: three red tests then one green

\qecho Expect fail -- nonsense
/*
	We use the "(select...)" in the argument list to avoid hard-coding IDs,
	this will make it easier to extend the tests further, if necessary.

	I didn't bother to assign the "red" values into variables in this section, 
	since we are only using each value once.
*/
select public.ssn_set((select id from public.people where lastname = 'Programmer'), 'unmitigated nonsense'::ssn_t, true);
select * from public.people where lastname = 'Programmer';

\qecho Expect fail -- too short
select public.ssn_set((select id from public.people where lastname = 'Programmer'), '01234567'::ssn_t, true);
select * from public.people where lastname = 'Programmer';

\qecho Expect fail -- too long
select public.ssn_set((select id from public.people where lastname = 'Programmer'), '0123456789'::ssn_t, true);
select * from public.people where lastname = 'Programmer';

-- using variables with psql makes it easier to change up the tests later
\set test_ssn 012345678
\set test_ssn2 987654321
\qecho Expect success -- just right
select public.ssn_set((select id from public.people where lastname = 'Programmer'), :'test_ssn'::ssn_t, true);
select * from public.people where lastname = 'Programmer';

\qecho Second round of testing on the social security function: one red and one green
\qecho Expect fail:  we have already used this SSN
insert into people(lastname) values ('Programmer Junior');
select public.ssn_set((select id from public.people where lastname = 'Programmer Junior'), :'test_ssn'::ssn_t, true);
select * from public.people where lastname = 'Programmer Junior';

\qecho Expect success: give Junior his/her own SSN
select public.ssn_set((select id from public.people where lastname = 'Programmer Junior'), :'test_ssn2'::ssn_t, true);
select * from public.people where lastname = 'Programmer Junior';

-- cleanup:  you have to back out of the sample database and then remove first it, then the role
\qecho A clean database is a happy database

\c postgres postgres
drop database sample;

drop role sample;

Now with more bugs: Debugging with PostgreSQL at FOSSCon 2019 – 8/17/2019

I am giving my Debugging With PostgreSQL talk tomorrow at FossCon. FOSSCon is the annual Free & Open Source Software Convention held every year in Philadelphia.

This version is lightly revised from last month’s version; added back in a few slides that I had to skip last time (I had 40 minutes last month, but 50 minutes tomorrow). And I fed back into the talk a bit of the audience feedback: more of what worked, less of the other stuff.

FOSSCon is fun, with a lot of great talks scheduled on Open Source & related. And it is free (donations are requested but not required.) Be seeing you.

Debugging with PostgreSQL – A Strategic (& Streamlined) Approach

Most popular slide at the talk: and the audience got all of them! (not counting the bit about the official name of Bangkok)

As planned, I gave a talk on Debugging with PostgreSQL at the Philly PostgreSQL conference at Wharton this last Friday (7/19/2019).

Went well: debugging is a great subject & I definitely struck a nerve with the audience; after the talk people were saying they knew about some of the points — which gave them some confidence — and others were new — which gave them some tools. Good.

My most popular slide was a quiz: only 10 lines of code — and from the PostgreSQL man page on foreign keys — but still three bugs. For the record, they are:

  • All of the data types should be domains, not physical types, so the city type should be something like “city_t”, defined as varchar(80). And the temperature should be, say, “fahrenheit_t” (or “celsius_t”), so you know what the units are.
  • The use of key words, like “date”, for field names is not great technique. It is ambiguous at best; breaks stuff at worst.
  • And the width for the city is way too small. Consider the name of Bangkok in Thai, the language of Bangkok: Krungthepmahanakhon Amonrattanakosin Mahintharayutthaya Mahadilokphop Noppharatratchathaniburirom Udomratchaniwetmahasathan Amonphimanawatansathit Sakkathattiyawitsanukamprasit. 177 characters! If you make the city’s type a domain, then you can revise the domain to be, say, “text” — and automagically get the type fixed everywhere you have a city reference.

I was scheduled to go late morning but went first because the opening speaker was still at his hotel. As a result I had the pleasant experience of hearing several later speakers refer to points made in my talk. The most popular was the phrase “lie consistently“.

I had built a form to collect Social Security numbers when I was at Bellcore (now Telcordia). It blew up when one fellow put in a variety of SSNs. I asked him what was going on. He said “I don’t want Bellcore to have my SSN. They have no legal right to it!”. “Fine by me, but just do me a favor & lie consistently“. We both left happy.

I did a run thru of the talk Sunday with my OTC (Official Talk Consultant); she pointed out, with her usual correctness, that I had tried to fit an entire software engineering course into 50 minutes. As a result, the early mornings & late evenings Monday thru Wednesday were spent reorganizing & rewriting. A 2nd run thru Wednesday evening went much better. OTC approved.

But when I did a final final talk & schedule check Friday morning I found the time blocks were now down to 40 minutes. Snip, snip, cut, cut, squeeze, squeeze. I cut out everything that wasn’t on message, useful, & fun. Definitely improved the talk. That which does not destroy us makes us strong. Or at least succinct.

Final version of the talk (PDF): Debugging with PostgreSQL — A Strategic (& Streamlined!) Approach.

WordPress Themes