pySpark映射多个变量

The code below maps values and column names of my reference df with my actual dataset, finding exact matches and if an exact match is found, return the OutputItemNameByValue. However, I'm trying to add the rule that when PrimaryLookupAttributeValue = DEFAULT to also return the OutputItemNameByValue.

The solution I'm trying out to tackle this is to create a new dataframe with null values - since there was no match provided by code below. Thus the next step would be to target the null values whose corresponding PrimaryLookupAttributeValue = DEFAULT to replace null by the OutputItemNameByValue.

  #create a map based on columns from reference_df
  map_key = concat_ws('\0', final_reference.PrimaryLookupAttributeName, final_reference.PrimaryLookupAttributeValue)
  map_value = final_reference.OutputItemNameByValue

  #dataframe of concatinated mappings to get the corresponding OutputValues from reference table
  d = final_reference.agg(collect_set(array(concat_ws('\0','PrimaryLookupAttributeName','PrimaryLookupAttributeValue'), 'OutputItemNameByValue')).alias('m')).first().m
  #display(d)

  #iterate through mapped values 
  mappings = create_map([lit(i) for i in chain.from_iterable(d)])

  #dataframe with corresponding matched OutputValues
  datasetPrimaryAttributes_False = datasetMatchedPortfolio.select("*",*[ mappings[concat_ws('\0', lit(c), col(c))].alias(c_name) for c,c_name in matchedAttributeName_List.items()]) 
  display(datasetPrimaryAttributes_False)

参考df

+--------------+----------------+--------------+--------+-----------------+--------------------+--------------------------+----------------------------+-------------------+-----------------------+---------------------------+-----------------------------+-------------------+---------------------+---------------------+
|SourceSystemId|SourceSystemName|     Portfolio|DomainId|       DomainName| PrimaryLookupEntity|PrimaryLookupAttributeName|SecondaryLookupAttributeName|StandardDomainMapId|StandardDomainItemMapId|PrimaryLookupAttributeValue|SecondaryLookupAttributeValue|OutputItemIdByValue|OutputItemCodeByValue|OutputItemNameByValue|
+--------------+----------------+--------------+--------+-----------------+--------------------+--------------------------+----------------------------+-------------------+-----------------------+---------------------------+-----------------------------+-------------------+---------------------+---------------------+
|             4|          ABC123|ALL_PORTFOLIOS|  100022|LeaseRecoveryType|ABC123_FF_Leases.csv|      LeaseRecoveryType |                            |                  9|                     329|                 Gross-modified|                               |             15|                     |            Modified Gross|
|             4|          ABC123|ALL_PORTFOLIOS|  100022|LeaseRecoveryType|ABC123_FF_Leases.csv|         LeaseRecoveryType|                            |                  9|                    330|             Gross         |                             |                 11|                     |                Gross|
|             4|          ABC123|ALL_PORTFOLIOS|  100022|LeaseRecoveryType|ABC123_FF_Leases.csv|         LeaseRecoveryType|                            |                  9|                    331|                 Gross w/base year|                      |                 18|                     |       Modified Gross|
|             4|          ABC123|ALL_PORTFOLIOS|  100011|      LeaseStatus|ABC123_FF_Leases.csv|               LeaseStatus|                            |                 10|                   1872|                  Abandoned|                             |                 10|                     |               Active|
|             4|          ABC123|ALL_PORTFOLIOS|  100011|      LeaseStatus|ABC123_FF_Leases.csv|               LeaseStatus|                            |                 10|                    332|                 Terminated|                             |                 10|                     |           Terminated|
|             4|          ABC123|ALL_PORTFOLIOS|  100011|      LeaseStatus|ABC123_FF_Leases.csv|               LeaseStatus|                            |                 10|                   1873|                    Archive|                             |                 11|                     |              Expired|
|             4|          ABC123|ALL_PORTFOLIOS|  100011|      LeaseStatus|ABC123_FF_Leases.csv|               LeaseStatus|                            |                 10|                    333|                 DEFAULT   |                             |                 10|                     |              Pending|
+--------------+----------------+--------------+--------+-----------------+--------------------+--------------------------+----------------------------+-------------------+-----------------------+---------------------------+-----------------------------+-------------------+---------------------+---------------------+
#dataframe with corresponding matched OutputValues
datasetPrimaryA_Nulls_False =  datasetPrimaryAttributes_False.where(reduce(lambda x, y: x | y, (f.col(x).isNull() for x in datasetPrimaryAttributes_False.columns)))
#display(datasetPrimaryA_Nulls_False)
datasetPrimaryA_Nulls_False.show()

+----------------+----------------+-------------+----------------+-----------+-----------------+---------------+--------------+----------------------+------------------------------+------------------------+
|SourceSystemName|       Portfolio|SourceLeaseID|SourcePropertyID|LeaseStatus|LeaseRecoveryType|      LeaseType| PortfolioRule|ConformedLeaseTypeName|ConformedLeaseRecoveryTypeName|ConformedLeaseStatusName|
+----------------+----------------+-------------+----------------+-----------+-----------------+---------------+--------------+----------------------+------------------------------+------------------------+
|          ABC123|ABC123 Portfolio|         4265|            1892|      Draft|             null|Field Associate|ALL_PORTFOLIOS|                   N/A|                          null|                 Pending|
|          ABC123|ABC123 Portfolio|         4266|            1893|      Draft|             null|Field Associate|ALL_PORTFOLIOS|                   N/A|                          null|                 Pending|
|          ABC123|ABC123 Portfolio|         1676|            1894| Terminated|             null|  Expense Lease|ALL_PORTFOLIOS|                Tenant|                          null|              Terminated|
|          ABC123|ABC123 Portfolio|         4304|            1937|      Draft|             null|Field Associate|ALL_PORTFOLIOS|                   N/A|                          null|                 Pending