Hello team,I have attended the webinar Databricks Certification Overview Series- Data Engineer on Jan 17Completed the Databricks Lakehouse fundamentals accreditation and Completed the survey.As per communication it is expected that I will receive Dat...
Hi @Indika Debnath​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...
Hi,Looking at the delta tables and how they are stored I have a question. If the delta tables are stored as parquet files in ADLS why if I copy/paste/rename a folder that corresponds to an existing table in the same location/database this does not ge...
@Sara Corral​ :When you copy/paste/rename a folder that corresponds to an existing delta table in the same location/database, it does not generate a copy of the previous table because delta tables are not just plain parquet files. They have additiona...
Hello All,Could anyone please suggest impact of Autoscaling in cluster cost ?Suppose if I have a cluster where min worker is 2 and max is 10 but most of the time active worker are 3 so the cluster will be billed for only 3 workers or for 10 worker(...
@Deepak Bhatt​ :Autoscaling in Databricks can have a significant impact on cluster cost, as it allows the cluster to dynamically add or remove workers based on the workload.In the scenario you described, if the active worker count is consistently at ...
Hello All,Could any one please suggest what is the best way to populate(Upsert) data from delta table into the sql server table.we are transforming our data in Databricks and storing data into the delta table. but for reporting purpose we need to pop...
@Deepak Bhatt​ :Yes, using the Spark Synapse connector could be a good option for upserting data from a Delta table into a SQL Server table. The Spark Synapse connector allows you to read and write data from Azure Synapse Analytics, formerly known as...
Using the install_libraries API, I installed a custom Python whl file on a running cluster. For certain types of requests, we have a requirement to install a different version of the same custom whl file in the running cluster. My problem is that uni...
Hi @Priya K​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your ...
Hi All.. Need your help in this issue what i am facing. Currently we are using data bricks as a platform to build pipeline and execute our talend ETL sqls converted into the spark sql framework as we were facing issues in loading the history data int...
Free Training: Databricks Lakehouse FundamentalsThe demand for technology roles is only growing – it's projected that over 150 million jobs will be added in the next five years. Across industries and regions, this is translating to increased demand f...
Below are the steps we have implemented to login through SSO.1. We have setup SSO and are able to login into Databricks using IDP (SiemensID Authentication). 2. After successful authentication, we have done the configuration of redirecting user to da...
Hello @Kaniz Fatma​ @Debayan Mukherjee​ Thanks for the response.We have raised this issue with Databricks team internally. We have shared the details with team. I will post the solution once we find any breakthrough to resolve it.
py4j.security.Py4JSecurityException: Method public org.apache.spark.sql.streaming.DataStreamReader org.apache.spark.sql.SQLContext.readStream() is not whitelisted on class class org.apache.spark.sql.SQLContextI already disable acl for cluster using "...
Hi @Ravi Teja​,Just a friendly follow-up. Do you still need help? if you do, please share more details, like DBR version, standard or High concurrency cluster? etc
Hi Team , So last year I acquired SQL Analyst Associate badge and due for renew this Jan 2023 . However when checked in Databricks Academy couldn't find the course . So has it been retired or removed ? If exists can someone help me with the course d...
I have a streaming pipeline that ingests json files from a data lake using autoloader. These files are dumped there periodically. Mostly the files contain duplicate data, but there are occasional changes. I am trying to process these files into a dat...
For clarity, here is the final code that avoids duplicates, using @Suteja Kanuri​ 's suggestion:import dlt
@dlt.table
def currStudents_dedup():
df = spark.readStream.format("delta").table("live.currStudents_ingest")
return (
df.drop...
Whenever using the displayHTML method or any python library that requires rendering HTML we get the following error in the results: Uncaught SyntaxError: Invalid or unexpected tokenWe cannot reproduce this error reliably, and resizing the html window...
Hi, If you could confirm the whole error stack will help us understanding the issue little clear. Also, please tag @Debayan​ with your next response which will notify me. Thank you!
I have the following error code in databricks when I want to unzip filesFileNotFoundError: [Errno 2] No such file or directory: but the file is there I already tried several ways and nothing worksI have tried modifying by placing/dbfs/mnt/dbfs/mnt/d...
So i am going to keep this generic as to all cloud provider storage options as its relevant across the board, (GCS, S3 and blob store). Nothing is mentioned in docs as far as i can see. Is there a use case against enabling object versioning in cloud ...
Hi @Matt User​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers you...
