- 8616 Views
- 5 replies
- 2 kudos
Get current date as string in databricks using scala
I want to get current date in scala as a string for example today current date is 3rd jan, want to store it as a new variable dynamically as below, how to get it.val currdate : String = “20220103”when I am using val currdate = Calendar.getInstance.ge...
- 8616 Views
- 5 replies
- 2 kudos
- 2 kudos
Hey @Vibhor Sethi Hope you are well!Thank you for posting your question and letting us know that you were able to resolve the issue. Would you be happy to mark it as the best solution? It would be really helpful for the other members too.Cheers!
- 2 kudos
- 4757 Views
- 2 replies
- 5 kudos
An error occurred while calling o303.mount: Operation failed: "This request is not authorized to perform this operation
Hi Team,We are unable to mount storage container in below scenario We created Gen 2 using VNet and added firewall restrictions (i.e allow trusted sources)And deployed Data bricks workspace with out VNet injection. Is it possible to add databricks pub...
- 4757 Views
- 2 replies
- 5 kudos
- 5 kudos
Hey @Sailaja B Hope everything is great!Does Hubert's response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?Thanks!
- 5 kudos
- 4216 Views
- 3 replies
- 1 kudos
Resolved! I can't access to my account.
I can't access to my account.This acccount was created today(not community, after 14 days trial it will chargable)when I'm try to access my account it gives meInvalid email address or passwordNote: Emails/usernames are case-sensitiveI tried to reset ...
- 4216 Views
- 3 replies
- 1 kudos
- 1 kudos
I got a reset link from the community. Actually the problem was with my username ,it did not identify a character within my username which was my email id.
- 1 kudos
- 4974 Views
- 1 replies
- 2 kudos
How to install JAR libraries from ADLS? I'm having an error
I mounted the ADLS to my Azure Databricks resource and I keep on getting this error when I try to install a JAR from a container:Library installation attempted on the driver node of cluster 0331-121709-buk0nvsq and failed. Please refer to the followi...
- 4974 Views
- 1 replies
- 2 kudos
- 21773 Views
- 6 replies
- 6 kudos
Resolved! Spark Driver Out of Memory Issue
Hi, I am executing a simple job in Databricks for which I am getting below error. I increased the Driver size still I faced same issue. Spark config :from pyspark.sql import SparkSessionspark_session = SparkSession.builder.appName("Demand Forecasting...
- 21773 Views
- 6 replies
- 6 kudos
- 6 kudos
I am getting the above issue while writing a Spark DF as a parquet file to AWS S3. Not doing any broadcast join actually.
- 6 kudos
- 4111 Views
- 1 replies
- 2 kudos
Resolved! Best way to encrypt PII data
Hi guys, I have around 600GB per load, in you opnion, what is the best way to encrypt PII data in terms of performance ? (lib, cluster type, etc.)Thank youWilliam
- 4111 Views
- 1 replies
- 2 kudos
- 2 kudos
Hello @William Scardua please check if the blog helps you.https://databricks.com/blog/2020/11/20/enforcing-column-level-encryption-and-avoiding-data-duplication-with-pii.html
- 2 kudos
- 4385 Views
- 1 replies
- 1 kudos
Facing mount/unmount issue while running same job parallelly with scala.
Using above configuration in cluster, when I run databricks job parallelly with multiple request at a same time, then I am getting mount/unmount issue. For an example : When I make three request to databricks job , it run 3 jobs parallelly but somet...
- 4385 Views
- 1 replies
- 1 kudos
- 1 kudos
hi @rahul upadhyay are you using the same mount path /mnt/rahul in all the 3 jobs? Could you please add the full error message?
- 1 kudos
- 19661 Views
- 8 replies
- 3 kudos
- 19661 Views
- 8 replies
- 3 kudos
- 12877 Views
- 3 replies
- 7 kudos
Resolved! Getting the error 'No such file or directory', when trying to access the json file
I am trying to write in my google sheet through Databricks but when it comes to reading the json, file containing the credentials, I am getting the error that No such file or directory exists.import gspread gc = gspread.service_account(filename='...
- 12877 Views
- 3 replies
- 7 kudos
- 7 kudos
Hi @Devarsh Shah The issue is not with json file but the location you are specifying while reading.As suggested by @Werner Stinckens please start using spark API to read the json file as below:spark.read.format("json").load("testjson")Please check ...
- 7 kudos
- 8599 Views
- 2 replies
- 5 kudos
Resolved! Write Empty Delta file in Datalake
hi all,Currently, i am trying to write an empty delta file in data lake, to do this i am doing the following:Reading parquet file from my landing zone ( this file consists only of the schema of SQL tables)df=spark.read.format('parquet').load(landingZ...
- 8599 Views
- 2 replies
- 5 kudos
- 5 kudos
Hi @bhagya s Since your source file is empty, there is no data file inside the centralizedZonePath directory i.e .parquet file is not created in the target location. However, _delta_log is the transaction log that holds the metadata of the delta for...
- 5 kudos
- 3684 Views
- 2 replies
- 0 kudos
How to merge delta data..
Data from Parquet to delta converted and delta files written into diff folders based on SRC_SYS_ID....Any one help me how to merge delta data from multiple folders.Regards.
- 3684 Views
- 2 replies
- 0 kudos
- 4522 Views
- 3 replies
- 2 kudos
How to read data from kafka topic using spark streaming
I have installed kafka-2.10-0.10.2. And using cluster with configuration: Runtime :6.4 Extended Support( scala 2.11,Spark 2.4.5) After this i am able to get mesgage son producer and consumer But when i try to read data from spark.readsttream and tr...
- 4522 Views
- 3 replies
- 2 kudos
- 14805 Views
- 4 replies
- 4 kudos
Getting error when using CDC in delta live table
Hi,I am trying to use CDC for delta live table, and when when I run the pipeline second time I get an error :org.apache.spark.sql.streaming.StreamingQueryException: Query tbl_cdc [id = ***-xx-xx-bf7e-6cb8b0deb690, runId = ***-xxxx-4031-ba74-b4b22be05...
- 14805 Views
- 4 replies
- 4 kudos
- 4 kudos
Hi @Palzor Lama,A streaming live table can only process append queries; that is, queries where new rows are inserted into the source table. Processing updates from source tables, for example, merges and deletes, is not supported. To process updates,...
- 4 kudos
- 9917 Views
- 5 replies
- 6 kudos
can we use spark-xml with delta live tables ?
Hiis there a way to use spark-xml with delta live tables (Azure Databricks) ?i 've try something like this without any succes for the momentCREATE LIVE TABLE df17 USING com.databricks.spark.xmlAS SELECT * FROM cloud_files("/mnt/dev/bronze/xml/s432799...
- 9917 Views
- 5 replies
- 6 kudos
- 6 kudos
This is a tough one since the only magic command available is %pip, but spark-xml is a maven package. The only way I found to do this was to install the spark-xml jar from the maven repo using the databricks-cli. You can reference the cluster ID usin...
- 6 kudos
- 1552 Views
- 0 replies
- 1 kudos
Databricks Office Hours Register for Office Hours to participate in a live Q&A session with Databricks experts! Our next events are scheduled for ...
Databricks Office HoursRegister for Office Hours to participate in a live Q&A session with Databricks experts! Our next events are scheduled for June 8th & June 22 from 8:00 am - 9:00am PT.This is your opportunity to connect directly with our experts...
- 1552 Views
- 0 replies
- 1 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
