Relatedness among tunes - is there a way to measure this?

Am looking for a measure that would group tunes into something like families? This could be fraternal twins like Coppers and Brass with Humours of Tullycreen, or non-twin siblings, such as Bucks Of Oranmore with Lucy Campbell, or Statia Donnally with Finbar’s Favorite. Other tunes might score low on ‘relationship’ with other tunes, such as The Blackbird (though there seems to be several versions of this - measure would have to factor in versions of the same tune), or Rights of Man.

It seems like, with the availability of ABC files on so many tunes, some smart individual(s) could find a way(s) to ‘score’ tunes as to their uniqueness.

Sounds like something the Dr. Dow would have been all over. https://thesession.org/members/4763/tunes
He was prolific when he was active on the tunes section here.

Try playing one of those pairings into Tunepal. I imagine the one you intended to play will come up with the highest confidence level. Now look at the runners up, the tunes Tunepal thought it could also be. Is your pairing listed there? Compare the confidence levels. Isn’t the ratio of them the "score" you were looking for?

I do find the lower percentage matches on TunePal fascinating. Often, I’ll realize that a related tune that I also know does have something in common with the tune I’m searching for, but had never considered it before.

If you want to know what’s going on "under-the-hood" with TunePal, here’s Dr. Bryan Duggan’s papers on it from 2010/2011:

https://arrow.tudublin.ie/cgi/viewcontent.cgi?article=1104&context=scschcomcon

https://arrow.tudublin.ie/cgi/viewcontent.cgi?article=1007&context=scschcomrep

Thanks for the valuable feedback!

It can be done but there you’d have to do a tremendous amount of prepping on the files before running any comparison routines. The routines themselves are very simple in unix/linux and could process the nearly 40,000 (currently) abc files here on the session in a very short time after the files met the comparison criteria. Getting the files into a suitable configuration is the problem. You’d have to make some decisions as to whether to convert triplets to a two-count note, ignore lead-notes, remove grace notes and so forth and so on. Fortunately, it wouldn’t affect the original score. There is a web site out there somewhere which at the moment I’m unable to remember at the moment that actually yields a relationship image and it’s pretty interesting. I suspect the owner of the sight would be unenthusiastic at having the abc contents of this site passed through it to dredge out the relationships though.

I’ve done some stuff like that personally but expanding that to a full-blown project is not something I or anyone else would undertake lightly.

I remember interviewing Breandan Breathnach way back in 1974 in the condemned school building in Dublin he had been given to conduct his work. He showed me how he simplified the first two bars of all the tunes he recorded from players to render them into (from memory) an 8 digit number, which he then used to be able to identify tunes with similar melodies but different names. As callison mentions above, it required setting some basic rules to deal with the twiddly bits.

That process could probably be coded these days. In Breandan’s time it was done on library cards! But I think we can go further and use whole tunes, rather than "incipits", as Breandan styled them. My suggestion above based on Tunepal would be such an approach.

What you’re looking for is basically this (or really the math behind it):
http://abcnotation.com/searchHelp#TuneGraph

Here’s an example of how it looks on the Kesh Jig:
http://abcnotation.com/tunePage?a=www.capeirish.com/webabc/working/source.folders/bsmi/bsmi.1/bsmi.1_ABC.txt/0110

The thing is that a simple comparison, such as using Tunepal with the confidence factor, or comparing text files with a text compare function, probably isn’t going to be sufficient. There does need to be some way to ignore or factor for things that we’d consider slight differences (GGG vs G3 for example).

http://msikio.online.fr/Breathnach/breandn.htm

I’ve recently made a program that parses all of my abc collection (>2500 tunes), splits them into parts, which it then compares against all other parts. It also compares all possible transpositions of the parts (using edit distance) to find those similarities too. Out comes a list of tune part similarities by percentage (a bit like the TunePal percentage).
Some of these are "false positives", but many of the similarities will make it into my next release of my abc collection as cross references.

@hnorbeck: When you state transpositions are you changing various settings of tunes all to the same key for comparison? Also, do you compare each part of a tune separately against other separate parts? There are a quite a few tunes where the first or second part may closely resemble another tune but the alternate part is far different (ignoring C, D, E parts etc). I suspect that in that case you’d have to determine the similarity percentage separately or suffer from the percentage never rising above 50%.

As an aside…
It’s just one "small" step from taking the comparison to finding the most common musical phrases per beat in each measure and using that information to create the most "basic" trad reel, jig etc. I’m certain that will eventually come to pass just as I am reasonably certain that a tune comprised of the most common elements of a genre would probably lack something undefinable/unattainable by computer analysis. I just did a quick search on the Tunes portion of thesession. There are no tunes with "Basic" in the name. I can see it now: "The Basic Reel", key of whatever key was most common. Perhaps useful for introduction to trad for beginners. Or the lazy. "I only know this [basic whatever] tune, can I join your session?". Arghh…

Sorry, rambling from Covid Boredom. :^(

If anyone wants a big data set to crunch through, there are weekly data dumps from The Session here:

https://github.com/adactio/TheSession-data

CSV, JSON, and SQLite.

My tunes search site (which, yes, I know I need to rewrite it and get it out of Flash, which only has a couple months of life left) uses a Levenshtein function for calculating the distance between typed input and every note interval of that length against every one of the tunes in my database… This is similar to the searching that TunePal is doing (minus the listening to the audio and auto transcribing it).

Something like that could be used, in theory, to build up families of tunes. But in practice, it’s quite a bit more difficult. You would probably want to compare the entire tunes, or maybe just the entire A and B parts, but the problem with that is really that there’s a ton of inconsistencies in how things are transcribed, whether the person put in the pickup notes, etc. So it might be a bit of a chore to normalize things before trying to compare large chunks of tunes. But then you could conceivably just run every tune against every other tune of that type and consider a distance score of over maybe 50 or 60 to be similar enough to be family members.

But there might be some other things to consider too, like what if you have 3 tunes, where two of them are similar enough to the third one, but not to each other… Or what about families of tunes in different time signatures. There are certainly tunes that are played as both jigs and reels, but there are even more tunes that don’t have quite that direct a connection, but they follow exactly the same chord progressions, etc.

Anyway, it would be an interesting thing to work on… if only someone would pay me lots of money to spend that much time doing it 😉

I immediately thought of your tunes search site, Reverend, and went to it to copy the url, and was saddened to discover it wouldn’t load on my recently updated Catalina OS.

Here’s a related thread to this one about "Tune taxonomy":
https://thesession.org/discussions/7768

Joe, if Safari on your latest Mac update has an issue with Reverend’s URL, can you try it on Chrome instead?

I think a thing that goes by the sound of a tune like Tunepal would give more sensible results than using ABC.

Unless there’s AI type software that can look at tunes rendered in ABC by their basic melodies rather than the specific ABC sequence.

I say that because you can write the same phrase in a dozen, probably two dozen, different ways that when played all sound virtually identical. Tunepal would recognise them as being the same tune, the ABC search engine on this site would give zero matches unless you happen to miraculously type in precisely the sequence that the contributor typed in.

I’ve typed in a dozen different ways of typing the same phrase here and consistently come up with zero matches even though I know very well the tune is here. When I finally find it- if I do- I find that the contributor has found a 13th way of notating that phrase.

An engine that can’t match the SAME tune certainly would be useless at matching related tunes.

BTW about relatedness of tunes, wouldn’t you have to consider the first parts and second parts as separate entities?

Because there’s loads of reels in A with quite different first parts, often not in the same key (some in A minor, some in A mixolydian, for example) which have virtually identical second parts.

Thing is, I think there are far more discrete first parts than second parts overall, in reels anyway.

"Sounds like something the Dr. Dow would have been all over."
What ever happened to him?

@callison: My algorithm makes all 12 possible transpositions and compares them. Transposing to a "standard key" is not feasible, since there are plenty of Irish tunes with ambiguous tonality.

Another preprocessing I do is remove all ornamentation before comparing.

I compare tune parts, not whole tunes. So if the B part of tune X is similar to the B part of tune Y I get a match. I want this for better cross referencing of my tune collection. I’m looking through all the similarities found (working from 100% downwards) "manually". I’ll probably give up around 65% where the similarities become too many.

Of course the same idea could be used for whole tunes.

Super interesting discussion. I’ve been thinking about this for years and wanting to give it a go. Thanks for all the useful links above and especially hnorbeck’s insights. Interesting that a normalised tune expressed in terms of intervals from a root won’t work because of challenges in defining "key" which is more fluid that key signatures might suggest.

The general topic of language/text analysis has interested me since a Markov matrix random text generator BASIC program I coded on a Commodore 64 back around ‘83.

Somewhere, directly or indirectly from links above, I ended up reading "CONSTRUCTING PROXIMITY GRAPHS TO EXPLORE SIMILARITIES IN LARGE-SCALE MELODIC DATASETS" by Chris Walshaw. Very interesting strategies and I laughed at the idea that his multi-level coarsening approach which simplifies the tune to the barest of notes that represent the essence of the melody… sounds like something Martin Hayes would do but without the science.

The Breathnach TuneID algorithm sounds like a musical variant of SOUNDEX used to categorise surnames by "sound" and was used by early USA census.

Very interesting discussion. Thanks all.

Very interesting. Actually if you go by the melodic contour, that can reveal relationships across tune-types, keys, mis-attributed keys and even national traditions. I’ve done this a couple of times but by poring over manuscripts rather than any other method - very intrigued by these possibilities for research though. Thanks all.