@Vidula Khanna​ I got this link for the certificate voucher register link.https://docs.google.com/presentation/d/1sy5hSSnFtncrpYY1EYi0WMsDkJK0dYk9iKBAeeAha8E/edit#slide=id.g1ade45a9cd6_0_543
Not sure if I'm missing something here, but running a task outside of a python function runs much much quicker than executing the same task inside a function. Is there something I'm missing with how spark handles functions? 1) def task(x): y = dostuf...
My observations show that timestamp difference has type of INTERVAL DAY TO SECONDS:select typeof(getdate() - current_date())
-----------------------------------------
interval day to secondBut is it guaranteed? Can it be DAY TO MINUTE or, say, YEAR T...
I have the following input: I am looking for a way to split the characters in the item_order_detail column into 2 columns itemID and itemName. As below output table uses SQL function in databricks with spark_sql version 3.2.1.Can someone suggest a so...
We are trying to read a column which is enum of array datatype from postgres as string datatype to target. We could able to achieve this by expilcitly using concat function while extracting like belowval jdbcDF3 = spark.read .format("jdbc") .option(...
I have no idea what I'm doing wrong. I synched this repo <https://github.com/databricks-academy/apache-spark-programming-with-databricks> as instructed. You can see that Notebook 4.1 is query optimization in the 'published' branch.https://github.com/...
It happened to me also in the past as courses are constantly updated. You can inform the training department through https://help.databricks.com/s/contact-us?ReqType=training
I'm working on the case to configure Kafka that is installed on my machine (Laptop) & I want to connect it with my Databricks account hosted on the AWS cloud.Secondly, I have CSV files that I want to use for real-time processing from Kafka to Databri...
For CSV, you need just to readStream in the notebook and append output to CSV using forEachBatch method.Your Kafka on PC needs to have the public address or you need to set AWS VPN and connect from your laptop to be in the same VPC as databricks.
Hello, I'm trying to export my dashboard to PDF but when I run the export, none of the data is exported, only blank visuals. I would appreciate any help in getting this resolved.Dashboard in DBX: Exported PDF:
Since yesterday, reading a file copied into the cluster is no longer working.What used to work:blob = gcs_bucket.get_blob("dev/data.ndjson") -> worksblob.download_to_filename("/tmp/data-copy.ndjson") -> worksdf = spark.read.json("/tmp/data-copy.ndjso...
I encountered this same issue, and figured out a fix!For some reason, it seems like only %sh cells can access the /tmp directory. So I just did...%sh ch /tmp/<file> /dbfs/<desired-location> and then accessed it form there using Spark.
Hi,Today I completed the test for Lakehouse fundamentals by scored 85%, still I haven't received the badge through my email francis@intellectyx.comKindly let me know please !-Francis
Hello Databricks Team,I completed my data engineer associate exam & passed the exam. But still now I haven't received certificate or badge. Also created help ticket, Ticket Number: #[00232273]​ Mail id : satishshravan888@gmail.comThanks,Satish
Hi @Satish Shravan​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...
Hi,I have been trying to deploy Access Connector resource on Azure using Azure Pipelines (YAML) and a Bicep template but I cannot find a solution to this error:ERROR: {"status":"Failed","error":{"code":"DeploymentFailed","message":"At least one resou...
Hi,I fixed this issue by adding the service principal to the list of service principals in the Account Console. My guess is that after the access connector is created an API call is made to the Databricks account and the service principal making that...
I have a set of Spark Dataframes that I convert into Temp Views to run Spark SQL with. Then, I delete them after my logic/use is complete. The delete step throws an odd error that I am not sure how to fix. Looking for some tips on fixing it. As a not...
Hi @Hubert Dudek​,​Pandas API doesn't support abfss protocol.You have three options:​If you need to use pandas, you can write the excel to the local file system (dbfs) and then move it to ABFSS (for example with dbutils)Write as csv directly in abfss...
