Check whether email column contains # and . using pyspark [closed] - apache-spark-sql

Question

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 3 years ago.
Improve this question
I have a data Frame that contains 8M data. There is one column name EMAIL contains Email address I have to check:
Email value must be of the form _#_._
Email value can only contain alphanumeric characters along with -_#.

Answer 1

In fact, there is a Python library designed for it, validate_email.
You can use the below code snippet to validate the email id.
from validate_email import validate_email
from pyspark.sql.types import BooleanType
from pyspark.sql.functions import udf
valid_email = udf(lambda x: validate_email(x), BooleanType())
emailvalidation.withColumn('is_valid', valid_email('EmailAddress')).show()
+--------------------+--------+
| email|is_valid|
+--------------------+--------+
|aswin.raja#gm.com | true|
| abc | false|
+--------------------+--------+
Another way is to use regular expressions. You can use the below code snippet.
import re
regex = '^\w+([\.-]?\w+)*#\w+([\.-]?\w+)*(\.\w{2,3})+$'
def check(email):
if(re.search(regex,email)):
print("Valid Email")
else:
print("Invalid Email")
if __name__ == '__main__' :
email = "aswin.raja#gm.com"
check(email)
email = "aswinraja.com"
check(email)
+--------+
|Valid |
|Invalid |
+--------+

Answer 2

You can use the below code to validate the Email Id in your table.
from pyspark.sql.functions import udf
from pyspark.sql.types import BooleanType
import re
def regex_search(string):
regex = '^\w+([\.-]?\w+)*#\w+([\.-]?\w+)*(\.\w{2,3})+$'
if re.search(regex, string, re.IGNORECASE):
return True
return False
validateEmail_udf = udf(regex_search, BooleanType())
df = df.withColumn("is_valid",validateEmail_udf(col("email")))

Answer 3

No need for UDF here, just use rlike function :
# not really the regex to validate emails but this handles your requirement
r = """^[\w\d-_\.]+\.[\w\d-_\.]+#[\w\d]+\.[\w\d]+$"""
df.withColumn("flag", when(col("email").rlike(regex), lit("valid")).otherwise(lit("invalid")))\
.show()
Gives:
+---+-----+----+-----------------+-------+
|ids|first|last| email| flag|
+---+-----+----+-----------------+-------+
| 1| aa| zxc|aswin.raja#gm.com| valid|
| 2| bb| asd|aswin.raja#gm.com| valid|
| 3| cc| qwe| aswinraja#ad.com|invalid|
| 4| dd| qwe|aswin.raja#gm.com| valid|
| 5| ee| qws| aswinraja#ad.com|invalid|
+---+-----+----+-----------------+-------+
For complete regex to validate email address check this post