Phil Moir's Blog

We have had quite a few releases since I last wrote, and I deeply apologise for this, and for not keeping members updated about progress in other areas of concern. This posting is going to focus on the changes we released this morning that are related to the search. I hope to post a blog later this week that will detail some of the other changes about all the other improvements we have made to the site since mid February and progress on other features.

Search Improvements

In the past, we have deliberately kept our searches simple, so that there is less confusion and to ensure that results are returned in a reasonable amount of time. We have been aware that this simplicity has created its own problems. Previous tinkering's with the search have usually led to a decline in performance, or less than accurate results, but we have listened to the requests to make some change. So the target for this release was to focus on BMD and Census searches, make more options available to search by, make more flexible options, return more accurate results, and to perform well. The whole team on GR has focused on making this change and we have been pleasantly surprised with our achievement, and in fact during development we found that we were able to introduce some options that previously we hadn't intended, but we felt they will be useful to members so kept them in.

Looking at "Search census records" first, the first thing we have added is an option on the Forename and Surname fields that lets you opt for either "Includes any of", "Exact match", "Starts with" or "Similar spelling". These options are replicated in all BMD and Census searches. The BMD search works slightly different and I will refer to these differences later. The "Exact match" means that we will search for the whole name provided and only return results with that name. Our aim is not to confuse these results by trying to predict what you want, but let you get straight to the specific results that matter. With regard to the Forename field, choosing "Exact match" will return those people who have ALL the forenames entered.

Search the census for Includes any of .. Forenames

Running the search above will return 2,648 results, with "Mary Barnes", "Ann Barnes", "Ann Sarah Barnes", etc. That's quite a lot to look through.

Results from the census for Includes any of .. Forenames

Now switch it to "Exact match" on Forename, and the result is quite different. There is 1 result. Also, the results came back in a few seconds or less. Sorting and moving through the pages of results is also significantly faster.

Results for the census for exact match .. Forenames

You can continue to make alterations to the search using the search box at the side of the results page, and for the Census records you can even run the same search by switching between Censuses.

Amend the search

The other options on name fields are "Starts with" and "Similar spelling". These options only check the first forename only. If more than one forename is entered, then it will check for Includes any of instead. So remember to only enter one Forename when using these options. Starts with is self explanatory, but Similar spelling requires a little guidance on expectations. This method uses Soundex which is a well known phonetic algorithm for indexing names by sound, in English. For a deeper understanding of this, read this Wikipedia article. Don't expect perfection with this method, but you may find it does find the ms-transcribed record your after. Also because of the complexity of this search, it does take a bit longer to perform. If we redo the search for "Mary Ann Barnes" with exact match on Forename and similar spelling on Surname, we get 291 matches. And here's a snippet of the results returned. As you can see it picks up amongst others, Byrnes, Burrington, Burnside, Burns, etc.

Similar spelling search

BMD searching is much the same although there are slight variations on the results returned. The BMD database does have a relevancy indicator, so we do try close matching, or return initialed results rather than just full name. Hyphenated surnames only need one part of the surname entered to be found. And the Includes any of forenames is not available as such because of the relevancy factor. Performance was a major concern on BMDs and one of the major contributors to the poor performance was the Place keywords option. We have changed this into a drop down selection by Country and County, which should ensure we can deliver results quickly.

Include other people in search

Across both the Census records and the BMDs we have added the ability to include forenames/surnames where relevant of other people that might be connected to the person searched for. For birth records, this is Mother's maiden name, for marriage records, this is Spouse's name (with variant options) and for census records, we have other forenames in household. These add complexity to the query so they will take a little longer to perform, but we hope this opens up new doors to finding your missing relations.

Lost Records - Found!

While we were testing our new search options we removed the mandatory Surname check, and by accident (or by intrigue), we wondered what would happen if we entered NO search criteria. You can try it here. There were 31,216 matches in the 1841 Census! These records have been transcribed, but because of the poor quality of the original document or image, the surname was not decipherable, so have been reported as ??? or blank or similar. So what we can now show is these hidden records for the first time on Genes. Even better, you can search using all the other options, Forename, Year of birth, Place keywords, etc. to narrow down your search and maybe that record you could never find before.

Lost records - no surname

Reference Details Shown and Searchable

On the Census transcription we now display the Reference Information. For years 1841 to 1901 this includes Folio, Page and Piece detail. For 1911, it includes the same plus 4 additional fields.

Reference information shown on transcription

This information can then be used on a new Search by reference option (Census only) for which a link appear below the standard Census search.

Search by reference link

This form requires that the core three fields be entered, but for 1911 it includes the additional fields also shown on the transcription. This result from this search will be individuals who either occupied the same building or were in similar location/area for which the census was taken.

Search by reference form

Previous Searches are back and better

You may not have noticed but the Recent Search feature was removed from the site for performance reasons earlier in the year. It used to store links to the last three searches you did, and this was done by storing a cookie on your local machine. If you cleared your cookies the information was lost, it also wasn't terribly accurate half the time and it only held three search references across the site. Not that useful. We dropped the use of cookies for this feature, and instead use a caching mechanism on our servers where we will store your searches for up to a year, so if your logged in, you can use different machines, but still see the previous search links. We also store up to 3 searches per recordset we have. i.e. 3 for 1911 census, 3 for 1901 census, etc. The link we store will re-perform the exact search, with all the same selections as you chose previously, not just the name and year. These links appear at the bottom of the relevant search form.

Recent searches

New "Year" validation on searches

Across all our searches and the Search All records feature we have implemented year validation. The intention is to save you time and deliver only relevant results. It is probably easier to show with an example. If I were to choose the following search.

Search year validation

We used to scan all our datasets looking for this person (Harry Turnbull) with this birth year (born 1880). Well obviously, he couldn't exist in the 1871 census or earlier as he wasn't born, so rather than checking and telling you there are no matches, we trim the results down to only those relevant sets of data. We will still tell you when no results have been found for valid searches, but we won't waste time checking or telling you about the others.

Only showing valid results

I hope this explanation will tempt you to try these new search features and help you to locate the records your after. We did have a few teething issues this morning, such as performance on the 1901 search, or searching using place keywords and other forenames, but we fixed these and have been monitoring usage, and all seems to be working well now. Do let us know if you like the new options and if they help you find records you hadn't been able to locate before.

Best regards, Phil Moir
Genes Reunited, Lead Developer