If you use HTML you can just assume that the images folder is at the root of the site (make sure to set the class to "reference").

<a class="reference" href="/images/eyeball-man.webp" alt="Eyeball Man"><img src="/images/eyeball-man.thumbnail.webp"></a>

It took me a while to stumble upon this, but there is an image shortcode for those of us not using ReST (although it isn't directly linked to in the documentation for the ReST directive, from what I could tell). So if you want to do it the easier way:

• Put the original image in the images folder
• use the shortcode:

{{% thumbnail "/images/eyeball-man.webp" alt="Eyeball Man" title="Eyeball Man" %}}{{% /thumbnail %}}

This will give you a thumbnail which will produce a pop-up to the original image when it's clicked. In this case the original pencil doodle was about the size of the end of my thumb so there's no real yount to the pop-up. I just used the image to have something to experiment with.

This is just to take a look.

For some reason pandas is acting up here so I had to separate the building of the dataframe and re-running the code blocks a few times. It even said that pandas couldn't find html5lib, which is kind of weird since the earlier read_html call worked…

URL = "https://www.wikiwand.com/en/List_of_countries_and_dependencies_by_population"
browser.get(URL)

Now that I think about it, maybe it takes some time for the javascript to render so you need to have a pause, especially if the internet connection is acting up.

tables = pandas.read_html(browser.page_source)

by_population = tables[1]
print(TABLE(by_population.head(1)))

Pandas doesn't seem to be able to handle the "% of world" column - probably because they put the percent symbol in the rows for some reason.

population_2 = by_population[Columns.population].sum()
print(f"{world[Columns.population] - population_2:,}")
print(f"{cleaned[Columns.population].sum() - population_2:,}")

-7,693,448,437
-7,861,884,121

What?

print(f"{population_2:,}")

15,507,529,605

According to the Wikipedia page the world population is 7,814,267,000 - so something isn't right here.

print(len(by_population))
print(len(cleaned))

242
252

They also don't have the same number of countries (it should be off by 1 because of Antarctica, not by 10).

countries = set(cleaned[Columns.country_cleaned])

# the new table doesn't have a space between Country and the parenthesis
countries_2 = set(by_population["Country(or dependent territory)"])
print(countries - countries_2)
print()
print(countries_2 - countries)

{'Northern Cyprus[note 4]', 'Western Sahara[note 12]', 'Taiwan', 'British Virgin Islands (United Kingdom)', 'Morocco', 'Bonaire (Netherlands)', 'Jersey (United Kingdom)', 'Cyprus', 'Anguilla (United Kingdom)', 'Isle of Man (United Kingdom)', 'Russia[note 11]', 'Wallis & Futuna (France)', 'Saba (Netherlands)', 'United States', 'Transnistria[note 3]', 'Vatican City[note 1]', 'Somaliland[note 8]', 'United States Virgin Islands (United States)', 'France', 'Turks and Caicos Islands (United Kingdom)', 'Cook Islands (New Zealand)', 'Democratic Republic of the Congo', 'Mayotte (France)', 'Tanzania', 'China', 'Guam (United States)', 'Falkland Islands (United Kingdom)', 'Guernsey (United Kingdom)', 'Northern Mariana Islands (United States)', 'Netherlands', 'Serbia', 'Republic of the Congo', 'Pitcairn Islands (United Kingdom)', 'Timor-Leste', 'Kosovo[note 2]', 'Eswatini (Swaziland)', 'Niue (New Zealand)', 'Denmark', 'Gibraltar (United Kingdom)', 'Saint Helena, Ascension and Tristan da Cunha (United Kingdom)', 'Finland', 'Pakistan', 'American Samoa (United States)', 'Sint Eustatius (Netherlands)', 'Bermuda (United Kingdom)', 'India', 'Abkhazia[note 6]', 'United Kingdom', 'Cayman Islands (United Kingdom)', 'Puerto Rico (United States)', 'Moldova', 'French Guiana (France)', 'South Ossetia[note 9]', 'Guadeloupe (France)', 'Ukraine [note 5]', 'Georgia', 'Svalbard and Jan Mayen (Norway)', 'Somalia', 'Tokelau (New Zealand)', 'Réunion (France)', 'Uruguay[note 7][clarification needed]', 'Martinique (France)', 'Federated States of Micronesia', 'Montserrat (United Kingdom)', 'Artsakh[note 10]', 'Antarctica'}

{'Pakistan[e]', 'India[c]', 'DR Congo', 'Falkland Islands (UK)', 'United Kingdom[h]', 'Wallis and Futuna (France)', 'Puerto Rico (US)', 'East Timor', 'Republic of Artsakh[z]', 'China[b]', 'Taiwan[l]', 'South Ossetia[aa]', 'Tokelau (NZ)', 'World', 'Ukraine[j]', 'Netherlands[m]', 'Saint Helena, Ascensionand Tristan da Cunha (UK)', 'American Samoa (US)', 'Tanzania[i]', 'Finland[q]', 'Vatican City[ab]', 'U.S. Virgin Islands (US)', 'Morocco[k]', 'Transnistria[w]', 'Somalia[n]', 'Abkhazia[y]', 'Cayman Islands (UK)', 'Gibraltar (UK)', 'Serbia[o]', 'Uruguay', 'Cook Islands (NZ)', 'Northern Cyprus[x]', 'Eswatini', 'Guernsey (UK)', 'Congo', 'Russia[f]', 'Bermuda (UK)', 'Pitcairn Islands (UK)', 'Guam (US)', 'Montserrat (UK)', 'Moldova[s]', 'Kosovo[t]', 'Denmark[p]', 'Jersey (UK)', 'France[g]', 'Isle of Man (UK)', 'Cyprus[u]', 'Anguilla (UK)', 'Georgia[r]', 'Western Sahara[v]', 'Northern Mariana Islands (US)', 'F.S. Micronesia', 'United States[d]', 'Turks and Caicos Islands (UK)', 'Niue (NZ)', 'British Virgin Islands (UK)'}

Some of that is just their notes being added, and an inconsistency of naming, e.g. United Kingdom vs UK. but the population only set also seems to break up things a little differently - they consider the countries within the Netherlands to be separate, for instance. This is messier than I thought, and I don't know how to reconcile the population explosion.

Okay, I just noticed something.

print(TABLE(by_population.tail(1)))

They copied the header here too, but pandas was able to parse the numbers (probably because the headers only have one row).

world_2 = by_population.iloc[-1]
by_population = by_population[:-1]

population_2 = by_population[Columns.population].sum()
print(f"{world[Columns.population] - population_2:,}")
print(f"{cleaned[Columns.population].sum() - population_2:,}")
print(f"{population_2:,}")

121,019,395
-47,416,289
7,693,262,605

The second table is short too, but it still has more than the original table… Maybe I just don't care about the numbers, as long as they're rankable. Although the fact that the tables have different numbers of countries isn't reassuring…

I think I'll see if making a combined table myself would be better. But first let's see what percentage of the population is missing.

print(f"By Population Table: {(by_population.Population/world_2.Population).sum() * 100:0.2f} %")
print(f"By Density Table: {(cleaned.Population/world_2.Population).sum() * 100:0.2f} %")

By Population Table: 98.45 %
By Density Table: 97.84 %

URL = "https://www.wikiwand.com/en/List_of_countries_and_dependencies_by_area"
browser.get(URL)

tables = pandas.read_html(browser.page_source)

by_area = tables[0]
print(TABLE(by_area.head(1)))

So once again we have a summation in there, but we have a bigger problem in that they combine the units in the same columns…

print(len(by_area))

264

Okay, so now we have another problem in that all three sets have differing numbers of entries. I'm guessing that trying to reconcile the country names is going to be a nightmare so I think I'll have to stick with the original table. I just looked at the Wikipedia page and this table includes "unrecognized states" which is probably why it has more countries. Maybe it's these unrecognized states that makes up the shortcoming in the population and area. Maybe.

males = table[table.columns[1:3]]
females = table[table.columns[3:]]
print(males.head(1))

   Males         
    Name   Number
0  James  4764644

Before we can plot them we have to change the count column to be (you guessed it) a number since that last row with the source information prevents pandas' ability to convert the column itself.

names.loc[:, "count"] = names["count"].astype(int)

plot = names.hvplot.bar(x="rank", y="count", by="gender").opts(
    title="Top Male and Female Names of the Last 100 Years",
    width=990,
    height=780,
    fontscale=2,
)

outcome = Embed(plot=plot, file_name="gender_plot")()

print(outcome)

Well, pretend that you don't see the x-ticks… It's interesting that the highest ranked male names have a higher count than the highest ranked female names.

URL = "https://en.wikipedia.org/wiki/List_of_legendary_creatures_by_type"

browser.get(URL)

This page is a little odd in that it uses tables but the tables then contain unordered lists which seem to give pandas a hard time, so I'll have to poke around a little bit using Beautiful Soup first to figure it out.

soup = BeautifulSoup(browser.page_source)

Just let selenium render the JQuery tables for us.

URL = "https://en.wikipedia.org/wiki/List_of_animal_names"
browser.get(URL)

After giving up on Requests-HTML I was using Beautiful Soup to pull out the table so I could give it to pandas, but then I looked at the pandas documentation and it turns out they already do this, so you don't need to, but you do need to know how many tables there are and which one you want (it returns a list of dataframes). In this case I want the last one.

table = pandas.read_html(browser.page_source)[-1]
print(TABLE(table.head(1)))

That's a little odd looking.

First, what's with all the A entries in the header?

print(table.columns)

MultiIndex([(                'Animal', 'A'),
            (                 'Young', 'A'),
            (                'Female', 'A'),
            (                  'Male', 'A'),
            (       'Collective noun', 'A'),
            (  'Collateral adjective', 'A'),
            ('Culinary noun for meat', 'A')],
           )

Hmm..

table.columns = [column[0] for column in table.columns]
print(table.columns)

Index(['Animal', 'Young', 'Female', 'Male', 'Collective noun',
       'Collateral adjective', 'Culinary noun for meat'],
      dtype='object')

print(TABLE(table.head(1)))

At this point I don't know what I really want to do with this. I was originally going to just save the animal names but now the other stuff looks kind of interesting. As an aside I had to look up Collateral Adjective on wikipedia:

A collateral adjective is an adjective that is identified with a particular noun in meaning, but that is not derived from that noun.

Good to know.

meats = table.dropna(subset=["Culinary noun for meat"])
print(len(meats))

49

So, now I noticed something else wrong.

print(TABLE(meats.iloc[-1:]))

Oops. Maybe I should have looked at the table more closely.

Let's see if using Beautiful Soup will help clean up the dataset a little.

soup = BeautifulSoup(browser.page_source)
soup_table = soup.find_all("table")[-1]

The read_html method always returns a list, even though I'm only passing in one table.

table = pandas.read_html(str(soup_table))[0]
print(TABLE(table.iloc[-1:]))

Actually, that looks worse.

I went back and looked at the page and the table is broken up by sub-headers indicating the first letter of the animal's name.

print(str(soup_table).split("\n")[10])

<th colspan="7"><span id="A"></span><b>A</b>

So, maybe I can clean that up a little before creating the table - here's where Beautiful Soup comes in. First I'll find all the tags whose ID matches a letter of the alphabet and then I'll use the decompose method to destroy the grand-parent of that tag - decompose destroys both the tag it's called on and all the descendants of that tag so this will destroy the grand-parent (table-row tag), parent (table-header) and matching tag (span with the ID matching the letter) and it's child (the bold tag). I'm also calling smooth and encode to clean the tree up once the tags are destroyed, although, to be honest, I don't know if that's really needed here, it just seemed like a good idea.

for character in ascii_uppercase:
    element = soup_table.find(id=character)
    if element is None:
        print(f"Element for {character} not found")
    else:
        element.parent.parent.decompose()

soup_table.smooth()
soup_table.encode()
table = pandas.read_html(str(soup_table))[0]
print(TABLE(table.head(1)))

So, now our first row is for Aardvark instead of the letter A. And now back to the meats.

meats = table.dropna(subset=["Culinary noun for meat"])
print(len(meats))

24

There are twenty-four entries in our table that have a noun referring to the animal as meat. Let's see what they are.

meats = meats.rename(columns={"Culinary noun for meat": "Meat"})
for row in meats.itertuples():
    print(f"{row.Animal}: {row.Meat}")

Cattle[note 3] (list): beef veal
Chicken (list): poultry
Deer: venison humble (organ meat)
DogfishAlso see Shark: Spiny dogfish capeshark (USA) flake (UK, AUS) huss (UK) kahada (CAN)[50] rigg (UK)rock salmon (UK)
Duck (list) Also see Mallard: poultry
Elk (wapiti): venison
Gaur: garabeef
Goat (list): chevon cabrito mutton
Goose: poultry
Guinea fowl: poultry
Guinea pig: cuy
MallardAlso see Duck: poultry
Moose: venison
Peafowl: poultry
Pig (list) Also see Boar: pork ham bacon
Pigeon (list): squab
RamAlso see Sheep: lamb mutton
Red deer: venison
Shark: flake (AUS)
Sheep (list) Also see Ram: lamb mutton hogget
Snail: escargot
Turkey (list): poultry
Water buffalo: carabeef
Whale: blubber

So now there's another problem - there are footnotes and links that makes things a bit messy. We could just clear out all link references in the HTML but that would destroy the animal names which are themselves links so I'll just do some string substitution in pandas instead to erase them.

FOOTNOTES = r"\[\w*\s*\d\]"
FOOTNOTES_2 = r"\s" + FOOTNOTES
EMPTY = ""
SPACE = " "
LIST = r"\(list\)"

for column in table.columns:
    table[column] = table[column].str.replace(FOOTNOTES_2, EMPTY)
    table[column] = table[column].str.replace(FOOTNOTES, SPACE)
    table[column] = table[column].str.replace(LIST, EMPTY)
print()
print(table.loc[32:32])

      Animal  Young Female  Male                       Collective noun  \
32  Cattle    calf    cow   bull  herd  drove  yoke (oxen) team (oxen)   

                                 Collateral adjective Culinary noun for meat  
32  bovine  taurine (male) vaccine (female) vituli...              beef veal  

meats = table.dropna(subset=["Culinary noun for meat"])
meats = meats.rename(columns={"Culinary noun for meat": "Meat"})
for row in meats.itertuples():
    print(f"{row.Animal}: {row.Meat}")

Cattle  : beef veal
Chicken : poultry
Deer: venison humble (organ meat)
DogfishAlso see Shark: Spiny dogfish capeshark (USA) flake (UK, AUS) huss (UK) kahada (CAN)  rigg (UK)rock salmon (UK)
Duck  Also see Mallard: poultry
Elk (wapiti): venison
Gaur: garabeef
Goat : chevon cabrito mutton
Goose: poultry
Guinea fowl: poultry
Guinea pig: cuy
MallardAlso see Duck: poultry
Moose: venison
Peafowl: poultry
Pig  Also see Boar: pork ham bacon
Pigeon : squab
RamAlso see Sheep: lamb mutton
Red deer: venison
Shark: flake (AUS)
Sheep  Also see Ram: lamb mutton hogget
Snail: escargot
Turkey : poultry
Water buffalo: carabeef
Whale: blubber

And now we have another prolbem - there's a weird smashing together of words (e.g. DogfishAlso) which appears to happen because wikipedia inserts <br/> tags to create a sub-row within the table rows and pandas appears to smash the rows together. Time to try again - I'm going to reload the soup and add a replace call to get rid of the breaks.

soup = BeautifulSoup(browser.page_source)
soup_table = soup.find_all("table")[-1]
for character in ascii_uppercase:
    element = soup_table.find(id=character)
    if element is None:
        print(f"Element for {character} not found")
    else:
        element.parent.parent.decompose()

for tag in soup_table.find_all("br"):
    tag.replace_with(SPACE)

soup_table.smooth()
soup_table.encode()

table = pandas.read_html(str(soup_table))[0]
print(TABLE(table[table.Animal.str.startswith("Ram")]))

So that fixes the smashing together, time to get rid of the footnotes and links to lists.

print(table.loc[32:32])
for column in table.columns:
    table[column] = table[column].str.replace(FOOTNOTES_2, EMPTY)
    table[column] = table[column].str.replace(FOOTNOTES, SPACE)
    table[column] = table[column].str.replace(LIST, EMPTY)
print()
print(table.loc[32:32])

                   Animal     Young   Female  Male  \
32  Cattle[note 3] (list)  calf[31]  cow[32]  bull   

                               Collective noun  \
32  herd[11] drove[11] yoke (oxen) team (oxen)   

                                 Collateral adjective Culinary noun for meat  
32  bovine[note 4] taurine (male) vaccine (female)...              beef veal  

      Animal  Young Female  Male                       Collective noun  \
32  Cattle    calf    cow   bull  herd  drove  yoke (oxen) team (oxen)   

                                 Collateral adjective Culinary noun for meat  
32  bovine  taurine (male) vaccine (female) vituli...              beef veal  

And once again, the meats.

meats = table.dropna(subset=["Culinary noun for meat"])
meats = meats.rename(columns={"Culinary noun for meat": "Meat"})
for row in meats.itertuples():
    print(f"{row.Animal}: {row.Meat}")

Cattle  : beef veal
Chicken : poultry
Deer: venison humble (organ meat)
Dogfish Also see Shark: Spiny dogfish capeshark (USA) flake (UK, AUS) huss (UK) kahada (CAN)  rigg (UK) rock salmon (UK)
Duck  Also see Mallard: poultry
Elk (wapiti): venison
Gaur: garabeef
Goat : chevon cabrito mutton
Goose: poultry
Guinea fowl: poultry
Guinea pig: cuy
Mallard Also see Duck: poultry
Moose: venison
Peafowl: poultry
Pig  Also see Boar: pork ham bacon
Pigeon : squab
Ram Also see Sheep: lamb mutton
Red deer: venison
Shark: flake (AUS)
Sheep  Also see Ram: lamb mutton hogget
Snail: escargot
Turkey : poultry
Water buffalo: carabeef
Whale: blubber

This isn't quite the nice set that I was hoping for, but, oh well. Maybe just one more clean-up to put paretheses around the Also see statements and onward.

PATTERN = "Also see (?P<name>\w+)"
def see_also(match):
    name = match["name"]
    return f"(Also see {name})"

table["Animal"] = table.Animal.str.replace(PATTERN, see_also)

for column in table.columns:
    table[column] = table[column].str.strip()

meats = table.dropna(subset=["Culinary noun for meat"])
meats = meats.rename(columns={"Culinary noun for meat": "Meat"})
for row in meats.itertuples():
    print(f"{row.Animal}: {row.Meat}")

Cattle: beef veal
Chicken: poultry
Deer: venison humble (organ meat)
Dogfish (Also see Shark): Spiny dogfish capeshark (USA) flake (UK, AUS) huss (UK) kahada (CAN)  rigg (UK) rock salmon (UK)
Duck  (Also see Mallard): poultry
Elk (wapiti): venison
Gaur: garabeef
Goat: chevon cabrito mutton
Goose: poultry
Guinea fowl: poultry
Guinea pig: cuy
Mallard (Also see Duck): poultry
Moose: venison
Peafowl: poultry
Pig  (Also see Boar): pork ham bacon
Pigeon: squab
Ram (Also see Sheep): lamb mutton
Red deer: venison
Shark: flake (AUS)
Sheep  (Also see Ram): lamb mutton hogget
Snail: escargot
Turkey: poultry
Water buffalo: carabeef
Whale: blubber