I am working in a notebook in Microsoft Fabric. I have a regular expression to find numbers in parentheses at the end of a string. In "this is a sentence (123)" I want to match (123), yet I keep just matching on 123. I can get it to work in a spark sql cell in the same notebook, but it does not work within a python function that uses spark.sql to run the SQL (this is pyspark, I assume, I'm new at all this so I'm hoping I am using terminology correctly). I have created a simple dataset to test with in the CTE below.

If I run this, it only returns the "test 123" row, but I want IDs 1 and 3. If I take out the $ in the where clause regex, it will return the other rows with numbers but only extract 123 in the extraction column. I can't figure out how to properly escape the parentheses.

df = spark.sql("""
     with c as (
        select 1 as id, 'test (123)' as descr --want to match
        union all
        select 2 as id, 'test 123' as descr --the match I do get but don't want
        union all
        select 3 as id, 'test (884)' as descr --want to match
        union all
        select 4 as id, 'work.test' as descr
        union all
        select 5 as id, 'work(999)test' as descr
    )
    select id, descr, regexp_extract(descr, '(\\([0-9]+\\))$') as extraction
    from c
    where regexp_like(descr, '(\\([0-9]+\\))$')
""")

df.show()